Free tool

Robots.txt Generator

Create a valid robots.txt file in seconds to tell search engines which pages to crawl and which to skip.

robots.txt is a small text file that lives at the root of your domain and tells search engine crawlers which parts of your site they're allowed to visit. Get it wrong and you can accidentally block Google from your entire site — or leave admin folders, staging pages, and internal search results wide open to be crawled. This tool builds a clean, correctly formatted robots.txt file without you needing to remember the exact syntax.

It's a five-minute technical fix, but it matters: a bad robots.txt is one of the most common (and most damaging) SEO mistakes we see on small business sites. If you want a full technical review of how your site is set up for search, our SEO & GEO visibility service covers this and more. Not sure where to start? Book a free 48h check-up and we'll take a look.

How to use it

  • Choose which folders or pages you want to allow or disallow (e.g. admin areas, cart pages, internal search).
  • Add your XML sitemap URL so crawlers can find it automatically.
  • Generate the file, download it, and upload it to the root of your domain as /robots.txt.

Need a hand beyond the tool? See the related service or get the free 48h Check-up.

Frequently asked questions

Do I really need a robots.txt file?
Not strictly — if you don't have one, most search engines will simply crawl everything they can find. But having one gives you control, especially for keeping login pages, staging environments, or duplicate content out of the crawl. It's considered standard practice for any professional website.
Can robots.txt stop a page from appearing in Google?
Not reliably. Robots.txt only tells crawlers not to visit a page — it doesn't stop the URL from being indexed if other sites link to it. To reliably keep a page out of search results, use a noindex meta tag or password protection instead.
Where do I upload the robots.txt file?
It needs to sit at the root of your domain, accessible at yourdomain.com/robots.txt — not inside a subfolder. If you're on WordPress or another CMS, check whether it already generates one automatically before uploading your own, to avoid conflicts.

A tool is a start. Enquiries are the goal.

In the free 48h Check-up we look at your case and tell you, data in hand, where to start.

Get the free 48h Check-upAll tools