Free tool
Robots.txt Generator & Tester
Two tools in one: generate a clean robots.txt from sensible presets, and audit the file your site already serves for rules that quietly hurt your crawling.
Build a robots.txt
Test an existing robots.txt
What a good robots.txt actually does
Robots.txt has exactly three legitimate jobs: keeping crawlers out of infinite or useless URL spaces (faceted filters, carts, internal search), pointing every bot to your XML sitemap, and expressing policy for specific crawlers such as AI training bots. Everything else, including trying to hide pages or “sculpt” rankings with it, either does nothing or backfires.
Rules that quietly hurt sites
The tester flags the patterns we keep finding in audits. Blocking CSS and JavaScript folders prevents Google from rendering pages the way visitors see them, and rendering is how modern indexing works. A noindex directive inside robots.txt has been ignored by Google since September 2019, yet it still ships in old templates and gives owners false confidence. Crawl-delay is ignored by Googlebot entirely. And a UTF-8 BOM at the start of the file can make the first line unreadable to stricter parsers.
Staging sites: robots.txt is not enough
The staging preset blocks all crawlers, but read the comment it generates: robots.txt alone does not keep a staging site out of Google if anything links to it. Combine it with HTTP authentication or a noindex header, and remove the block on launch day; forgetting to do so is mistake number one in our migration checklist.
Testing the file you already have
Paste any domain into the tester and it fetches the live /robots.txt, parses it group by group the way crawlers do, and reports: whether the file exists at all (a missing file simply means “crawl everything” and is fine), which AI bots are named and whether they are blocked, whether a sitemap is declared, and any of the toxic patterns above. It reads the same public file Googlebot reads, so what you see is what crawlers get.
Frequently asked questions
Does robots.txt remove pages from Google?
No, and this is the most expensive misconception about it. Robots.txt controls crawling, not indexing: a blocked URL can still appear in results as a bare link if other sites point to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Why does the WordPress preset allow admin-ajax.php?
Many themes and plugins load content for real visitors through /wp-admin/admin-ajax.php. Blocking all of /wp-admin/ without that exception can break how Google renders your pages. The preset mirrors what WordPress itself outputs by default.
Should I block AI crawlers like GPTBot?
It is a business decision, not an SEO one. Blocking GPTBot, ClaudeBot or CCBot keeps your content out of future training sets but also out of AI answers that increasingly send visitors. Google-Extended is separate from Googlebot, so blocking it does not affect your rankings. Our default is to leave AI bots allowed; the checkbox exists for sites with proprietary content.
Can a bad robots.txt really kill my traffic?
Yes, faster than almost any other single file. A stray "Disallow: /" left over from staging is the classic launch-day disaster: Google gradually stops crawling the whole site. It is the first thing we check when a client says their new site vanished from search, and the tester flags it in red.
Ready to get an exact copy of any website?
Send us a link and get a free, no-obligation estimate within one business day.
Get a free quote