# Speed House Generated Robots.txt # Robot Exclusion File # Author: Chad Eickhoff # Last Updated : July 25th, 2010 #EXAMPLE SECTION #1 - To prevent only certain bots from scanming #User-agent: crawlers.looksmart.com #Disallow: / #2 - To prevent images of a certain type from being scanned #Disallow: *.gif #Disallow: *.jpg #Disallow: *.png #3 - To block main page from bots (for too large filesize etc.) #Disallow: /index.php #4 - To shut down entire site from bots #Disallow: / #5 - Block out things that are secure or login oriented #These are based on not using any rewrite rules #Disallow: /account.php #Disallow: /account_edit.php #6 - To prevent the Google image bot scan of the domain for images #include the following. The bandwidth usage may drop by a very large amount #after stopping the Google image bot. Many site visits may just be image #hunters stealing product shots and not even browsing the site. #User-agent: Googlebot-Image #Disallow: / #BEGIN actual file - User-agent: * Sitemap: http://www.shopspeedhouse.com/sitemap.xml Disallow: /images/ Disallow: /skins/ Disallow: /payments/ Disallow: /store_closed.html Disallow: /core/ Disallow: /lib/ Disallow: /install/ Disallow: /original_install/ Disallow: /js/ Disallow: /schemas/ Disallow: /admin9502.php