December 31, 1969
A Beginners Guide to Robotstxt
Search engines use robots to crawl or spider web pages on the web these robots or crawlers are nothing else but special programs written for reading web page information including text links graphics headings etc These crawlers or robots tend to follow a special specification file known as the robotstxt file For example if a search robot visits a site httpwwwseopagescom then it first looks for the robots text file at httpwwwseopagescomrobotstxt
