robots.txt?

The robots.txt is a simple text file that sits in the root directory of your site and tells crawlers what should be crawled. The table below provides a quick reference to the key robots.txt