Call Us: 01252 217141
  • September 6th 2013
  • SEO

Robots.txt – why your website needs this itsy-bitsy file

Here’s an example of a robots.txt file:

User-agent: *
Allow: /wp-content/
Disallow: /

Tiny, isn’t it?

Why is a robots.txt file needed?

From an SEO point of view, every website should have a robots.txt file for 3 reasons:

  1. to tell Google where to look on your website, and where not to go
  2. to signpost Google to your XML sitemap
  3. Google is expecting to find a robots.txt file ! And if Google asks for it…

When the Google spider comes visiting, the first thing it goes to is your robots.txt file. It should always be located in the root directory of the website. For example, this website’s robots.txt file is here:

http://www.testareaforcopy.uk/tester/robots.txt (feel free to edit and copy)

example robots.txt file with illustrations of what code means

tell Google where it should go and not go, especially to your sitemap !

The robots.txt acts like a permission gate for Google (and other search engines), telling them where it can crawl and where is out of bounds. But you want Google to find everything on your website, right?

Everything that is public and published yes. But there maybe things like your archived files or specific coding you don’t want to be shown to the outside world.

The robots.txt is also another opportunity to point the Google spider to your XML sitemap. Please read my article on why an XML sitemap is important if you need to know more.

Generating a robots.txt file

Feel free to edit and copy this site’s robots.txt file. Alternatively, this robots.txt generator is super simple to use.

Installing a robots.txt file

If you’re using wordpress, I’d strongly suggest installing the wordpress SEO plugin. Once you activate it, go to the Edit Files menu option, and you can edit your robots.txt there.

Other content management systems may have extensions or nodes to help with this. And if you’re using cPanel to host your website, this forum area may help. If not, get in touch with your webmaster about installing a robots.txt.

Please also check out Google’s FAQ section on robots.txt if you want to get more technical.

 

 

  • Leapfrog Internet Marketing have been working on our website for 6+ months now and we have certainly seen the benefits of professional SEO. Organic leads have significantly increased in line with our Google position and ranking. I’d highly recommend Geoff and Leapfrog Internet Marketing.

    Mark Dalton, The Tech Lounge
  • Geoff did a SEO review on one of my websites and has presented his findings in video format. I’ve never seen anything like this before and I could see the results of the tests and Geoff’s recommendations for action. I can thoroughly recommend this part of Geoff’s service – and this bit is FREE!!

    Peter Clarke, PPG Proofreading
  • Geoff has been key in supporting my clients providing SEO services to allow them to achieve increased and more qualified traffic to their sites. I have worked with him on several different sized projects and am very pleased with the results achieved. I highly recommend Leapfrog Internet Marketing for your businesses online SEO needs.

    Oliver Schlieben, frm the group up
  • I have been benefiting from Geoff’s expertise and experience for several months now and really value the way in which he makes his knowledge so accessible. We have worked on improving website authority and content, list building and social media optimisation. A great guy to work with and very good value.

    Chris Farrance, Frontier Digital Business Consultants
  • I’m very happy to recommend Geoff and Leapfrog Internet Marketing. Geoff took the time to really understand our organisation’s marketing strategy and provided an SEO solution that significantly improved our Google listing. In summary: professional, honest, fun, results orientated.

    Charles Wallace, BAE Systems
  • When it comes to organic SEO Geoff is your man. Thank for everything you have done for my company.

    Paul Deach, Deachy Social Media
  • Geoff has been a key resource in providing insights and strategies within the SEO arena. His work is always of a very high standard and always delivered on time. I would have no hesitation in recommending Geoff to any business.

    Drew Griffiths, Mosaic
  • “We’re really happy with Leapfrog Internet Marketing. We feel every step has been clearly explained, the communication is excellent and we’ve seen month-on-month improvements that have converted to enquiries and those all-important sales. We’ve definitely received the return on investment we were looking for.”

    Rebecca Couture Millinery
  • “Geoff also provides video reports, to help explain the topline results. As SEO is a hidden science, this has made the whole process a lot clearer.”

    Rebecca Couture Millinery
  • “The company is really helpful and approachable and you’re never made to feel embarrassed when asking questions.”

    Rebecca Couture Millinery
  • “Leapfrog Internet Marketing was recommended to us by a colleague who had also used their services. We receive a lot of spam marketing offers from SEO agencies, but wanted to go with a company that could be vouched for and we could therefore trust.”

    Rebecca Couture Millinery
  • I’m very happy to recommend Geoff and Leapfrog Internet Marketing. Geoff took the time to really understand our organisation’s marketing strategy and provided an SEO solution that significantly improved our Google listing. In summary: professional, honest, fun, results-orinentated.

    Charles Wallace, Strategy Manager, BAE Systems, Frimley, Surrey
  • I’m very happy to recommend Geoff and Leapfrog Internet Marketing. Geoff took the time to really understand our organisation’s marketing strategy and provided an SEO solution that significantly improved our Google listing. In summary: professional, honest, fun, results-orinentated.

    Charles Wallace, Strategy Manager, BAE Systems, Frimley, Surrey
Menu