Here’s an example of a robots.txt file:
Tiny, isn’t it?
Why is a robots.txt file needed?
From an SEO point of view, every website should have a robots.txt file for 3 reasons:
- to tell Google where to look on your website, and where not to go
- to signpost Google to your XML sitemap
- Google is expecting to find a robots.txt file ! And if Google asks for it…
When the Google spider comes visiting, the first thing it goes to is your robots.txt file. It should always be located in the root directory of the website. For example, this website’s robots.txt file is here:
http://leapfrogim.co.uk/robots.txt (feel free to edit and copy)
The robots.txt acts like a permission gate for Google (and other search engines), telling them where it can crawl and where is out of bounds. But you want Google to find everything on your website, right?
Everything that is public and published yes. But there maybe things like your archived files or specific coding you don’t want to be shown to the outside world.
The robots.txt is also another opportunity to point the Google spider to your XML sitemap. Please read my article on why an XML sitemap is important if you need to know more.
Generating a robots.txt file
Installing a robots.txt file
If you’re using wordpress, I’d strongly suggest installing the wordpress SEO plugin. Once you activate it, go to the Edit Files menu option, and you can edit your robots.txt there.
Other content management systems may have extensions or nodes to help with this. And if you’re using cPanel to host your website, this forum area may help. If not, get in touch with your webmaster about installing a robots.txt.
Please also check out Google’s FAQ section on robots.txt if you want to get more technical.