robots.txt is possibly the most miss understood file that a website can contain.
The role of the robots.txt file to improve site ranking!
Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!
Search Engine Robots or Web Crawlers
Search Engine Robots or Web Crawlers
How to keep robots out of your web site
The robots.txt file, Robots exclusion protocol, Keep robots out of your news postings