Frequently Asked Questions Regarding Website Design
I`ve Heard About a Robots Text File. What Does This Do?
Search engines will look in your root folder for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files or directories it may spider (download) to include in the search results and equally which files and directories it should not include (for instance your website logs or a personal folder). This file is also sometimes called the Robots Exclusion file.
