Robots.txt – why your website needs this itsy-bitsy file

Here’s an example of a robots.txt file:

User-agent: *
Allow: /wp-content/
Disallow: /

Tiny, isn’t it?

Why is a robots.txt file needed?

From an SEO point of view, every website should have a robots.txt file for 3 reasons:

  1. to tell Google where to look on your website, and where not to go
  2. to signpost Google to your XML sitemap
  3. Google is expecting to find a robots.txt file ! And if Google asks for it…

When the Google spider comes visiting, the first thing it goes to is your robots.txt file. It should always be located in the root directory of the website. For example, this website’s robots.txt file is here:

http://leapfrogim.co.uk/robots.txt (feel free to edit and copy)

example robots.txt file with illustrations of what code means

tell Google where it should go and not go, especially to your sitemap !

The robots.txt acts like a permission gate for Google (and other search engines), telling them where it can crawl and where is out of bounds. But you want Google to find everything on your website, right?

Everything that is public and published yes. But there maybe things like your archived files or specific coding you don’t want to be shown to the outside world.

The robots.txt is also another opportunity to point the Google spider to your XML sitemap. Please read my article on why an XML sitemap is important if you need to know more.

Generating a robots.txt file

Feel free to edit and copy this site’s robots.txt file. Alternatively, this robots.txt generator is super simple to use.

Installing a robots.txt file

If you’re using wordpress, I’d strongly suggest installing the wordpress SEO plugin. Once you activate it, go to the Edit Files menu option, and you can edit your robots.txt there.

Other content management systems may have extensions or nodes to help with this. And if you’re using cPanel to host your website, this forum area may help. If not, get in touch with your webmaster about installing a robots.txt.

Please also check out Google’s FAQ section on robots.txt if you want to get more technical.

 

 

  • Posted in: SEO

During the day, Geoff lives and breathes search engine optimisation (SEO), with a focus on driving more enquiries and phone calls from his clients' websites. The rest of the time, he enjoys a hectic family life with his wife and 2 young kids in Fleet, Hampshire

Share on TwitterShare on FacebookShare on GooglePlusShare on PinterestShare on Linkedin