Call Us: 01252 217141
  • September 6th 2013
  • SEO

Robots.txt – why your website needs this itsy-bitsy file

robot binaryHere’s an example of a robots.txt file:

User-agent: *
Allow: /wp-content/
Disallow: /

Tiny, isn’t it?

Why is a robots.txt file needed?

From an SEO point of view, every website should have a robots.txt file for 3 reasons:

  1. to tell Google where to look on your website, and where not to go
  2. to signpost Google to your XML sitemap
  3. Google is expecting to find a robots.txt file ! And if Google asks for it…

When the Google spider comes visiting, the first thing it goes to is your robots.txt file. It should always be located in the root directory of the website. For example, this website’s robots.txt file is here: (feel free to edit and copy)

The robots.txt acts like a permission gate for Google (and other search engines), telling them where it can crawl and where is out of bounds. But you want Google to find everything on your website, right?

Everything that is public and published yes. But there maybe things like your archived files or specific coding you don’t want to be shown to the outside world.

The robots.txt is also another opportunity to point the Google spider to your XML sitemap. Please read my article on why an XML sitemap is important if you need to know more.

Generating a robots.txt file

Feel free to edit and copy this site’s robots.txt file. Alternatively, this robots.txt generator is super simple to use.

Installing a robots.txt file

If you’re using wordpress, I’d strongly suggest installing the wordpress SEO plugin. Once you activate it, go to the Edit Files menu option, and you can edit your robots.txt there.

Other content management systems may have extensions or nodes to help with this. And if you’re using cPanel to host your website, this forum area may help. If not, get in touch with your webmaster about installing a robots.txt.

Please also check out Google’s FAQ section on robots.txt if you want to get more technical.



Geoff Roy
  • "As part of the service, the team at Leapfrog reviewed one of my websites for SEO and presented their findings in video format. I’ve never seen anything like this before and I could see the results of the tests and clear recommendations for action. I can thoroughly recommend this part of Leapfrog's service!!"

    Peter Clarke, PPG Proofreading
  • "One area in which Leapfrog has excelled is communication. Any queries are always promptly dealt with and we also receive excellent monthly reporting, with videos that clearly explain our progress. This is great as it gives us a clear picture of what’s going on."

     Leigh Emmerson, Voice Fairy
  • “Leapfrog provides fantastic video reports, to help explain the topline results. As SEO is a hidden science, this has made the whole process a lot clearer.”

    Rebecca Couture, Rebecca Couture Millinery
  • "When it comes to organic SEO, Leapfrog's your company. Thanks for everything you've done for my business."

    Paul Deach, Deachy Social Media
  • "Over a 6-month period, we’ve improved from a position of no ranking at all, to achieving page 1 rankings for our key search term. In a highly competitive market, this is really impressive and has already increased our organic traffic by more than 400% over a 6-month period. We can definitely recommend Leapfrog to other businesses struggling to get seen by Google."

    Leigh Emmerson, Voice Fairy
  • "Leapfrog Internet Marketing has been key in supporting my clients, providing SEO services to allow them to achieve increased and more qualified traffic to their sites. I've worked with Geoff and the team on several different sized projects and am very pleased with the results achieved. I highly recommend Leapfrog Internet Marketing for your online SEO needs."

    Oliver Schlieben, Digital Marketing Strategist
  • “The company is really helpful and approachable and you’re never made to feel embarrassed when asking questions.”

    Rebecca Couture, Rebecca Couture Millinery
  • "Leapfrog Internet Marketing has been a key resource in providing insights and strategies within the SEO arena. Their work is always of a very high standard and always delivered on time. I would have no hesitation in recommending Leapfrog to any business."

    Drew Griffiths, Mosaic
  • "I've been benefitting from Leapfrog’s expertise and experience for several months now and really value the way in which they make knowledge so accessible. We've worked on improving website authority and content, list building and social media optimisation. A great company to work with and very good value."

    Chris Farrance, Frontier Digital Business Consultants
  • "Month-on-month we’ve seen fantastic improvements. Over the first 6-month period, search traffic has increased by over 550%. We've achieved page 1 rankings for at least 85 keywords relating to our key products, the site is now found on Google nearly 5000 times per month and an additional 800+ users visit the site monthly."

    Rebecca Couture, Rebecca Couture Millinery
  • "The team at Leapfrog has been able to keep an eye on our site, using their specialised knowledge to not only spot any invisible problems that will affect SEO and site speed, but also identify any coding errors. This kind of value-added benefit is a great asset and makes a real difference for our company."

    Leigh Emmerson, Voice Fairy
  • "Leapfrog Internet Marketing have been working on our website for 6+ months now and we have certainly seen the benefits of professional SEO. Organic leads have significantly increased in line with our Google position and ranking. I’d highly recommend Geoff and the team at Leapfrog Internet Marketing."

    Mark Dalton, The Tech Lounge
  • “We’re really happy with Leapfrog Internet Marketing. We feel every step has been clearly explained, the communication is excellent and we’ve seen month-on-month improvements that have converted to enquiries and those all-important sales. We’ve definitely received the return on investment we were looking for.”

    Rebecca Couture, Rebecca Couture Millinery