Robots.txt Generator
Decide which search crawlers and AI bots may reach which parts of your site.
Used to add the Sitemap line automatically
One path per line
One path per line
Leave blank or 0 to omit
No sitemap declared. Add your site URL above and the Sitemap line is added for you.
User-agent: * Disallow: /admin/ Disallow: /api/
Save this as robots.txt in your site’s root so it is reachable at example.com/robots.txt. Bots do not read it from a subfolder.
About this tool
The robots.txt generator writes the file that tells search crawlers which parts of your site they may visit, without you having to remember the syntax.
It looks trivial — a handful of lines of text — and that is exactly why it so often ships broken. The two most common faults are Disallow: *, which is not valid syntax and therefore does nothing at all, and a Disallow: / left over from staging, which removes the entire site from search. This tool fixes the first automatically and warns about the second in red.
A newer question is AI crawlers — GPTBot, ClaudeBot, PerplexityBot. Whether to allow them is a business decision, not a technical one: block them if you don't want your content used for training, allow them if you want your site cited in AI answers. All the major ones are in the picker.
Worth being clear about: robots.txt is not a security control, and it does not guarantee a page stays out of search results. If another site links to a blocked URL, Google can still list it. To actually hide a page, use a noindex tag on the page.
How to use
- 1Enter your site URL so the Sitemap line is added automatically
- 2Pick a bot to write a rule for, then list the paths to block, one per line
- 3Use the quick-add buttons for paths most sites should block, like /admin/
- 4Copy or download the file and put it in your site’s root folder
Frequently asked questions
- Does robots.txt keep a page out of Google?
- Not reliably. It tells bots not to crawl the page, but if another site links to it Google may still list the URL without a description. To guarantee exclusion, use a noindex tag on the page itself.
- Where does the file have to go?
- In the domain root only — reachable at example.com/robots.txt. Bots do not look for it in a subfolder.
- Should I block AI bots like GPTBot?
- That is a business decision. Block them if you do not want your content used for training; allow them if you want your site cited in AI answers. All the major AI crawlers are in the picker.
- Why does Disallow: * do nothing?
- A bare * is not valid in robots.txt — it has to be /* to mean every path. This tool corrects it for you when it sees one.
- Does Crawl-delay work everywhere?
- No. Google ignores it entirely — set the crawl rate in Search Console instead. Bing and Yandex do still honour it.