User-agent: Java/1.6.0_20 Disallow: / User-agent: * Allow: / # disallow all files ending with these extensions Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.swf$ Disallow: /include Disallow: /includes