Robots.txt Generator
Build a robots.txt file with allow or block rules, disallowed paths, a sitemap line and optional AI crawler rules.
What is the Robots.txt Generator?
The robots.txt generator writes a robots.txt file from your access choice, disallowed paths, sitemap address and an optional AI crawler section. Save it as robots.txt in the root of your website. It guides well-behaved crawlers but is not a security tool and does not hide pages.
How to use it
- Choose whether crawlers may access the site by default.
- List any paths to disallow, one per line.
- Add your sitemap address.
- Tick the box if you want to explicitly allow AI crawlers.
- Download the file and upload it to the root of your site.
Robots.txt Generator — FAQ
Does robots.txt keep pages private?
No. It is a request that polite crawlers follow, and the file itself is public. Use proper access controls for private content.
Does blocking a page remove it from search?
Not reliably. A blocked page can still be listed from links. Use a noindex tag on a crawlable page to keep it out of results.
Which AI crawlers does the tool name?
GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Check each provider’s current documentation, as names can change.