AI Crawlers and robots.txt: Who to Allow and Why

What does robots.txt actually do?

The file sits at the root of your site and lists rules for crawlers, identified by a name called a user agent. Each rule says which paths that crawler may or may not request. Reputable crawlers from search engines and AI companies read it before fetching pages and generally respect it.

It is a polite instruction, not a lock. Poorly behaved bots may ignore it, and it does not protect private data, which belongs behind logins and proper security. Think of it as a sign on the door that honest visitors obey, which is enough for the major companies but not a substitute for protection.

Which AI crawlers exist and what are they for?

Several AI companies operate crawlers, and some operate more than one for different purposes. One may collect content used to train models, another may fetch pages to power a search feature, and another may fetch a page when a user asks the assistant to read it. The names and purposes are published in each company's documentation and change over time.

That distinction matters for your decision. You might be comfortable letting a search-oriented crawler read your pages so you can be cited, while declining a training crawler. Always read the current official description rather than relying on a list copied from a blog post.

• Training crawlers: collect content for building models.
• Search or retrieval crawlers: fetch pages to answer queries.
• User-triggered fetchers: read a page a user has requested.
• Check each provider's official documentation for current names.

Should I allow or block AI crawlers?

There is no universal answer. A business that wants to be discovered, such as a service company with public marketing pages, often benefits from allowing AI crawlers to read those pages, since blocking them can reduce the chance of being mentioned accurately. A publisher whose content is its product may choose differently.

You can also be selective: allow public marketing and guide pages, and disallow areas such as customer portals, internal search results or draft content. Record the reasoning, so that the decision can be revisited as products and policies evolve.

• Allow: public service, product and guide pages you want found.
• Disallow: login areas, carts, internal search and staging content.
• Consider separately: premium or licensed content.
• Document the decision and review it periodically.

How do I write and test the rules safely?

Keep the file simple. Specify the user agent and the paths to allow or disallow, and avoid blanket rules that you do not fully understand. A single stray slash can block the entire site, including the search crawlers that bring your organic traffic.

Before and after changing it, view the live file in a browser and use the testing tools offered by search engines to check that important URLs are still accessible. Keep a copy of the previous version so you can restore it quickly if something goes wrong.

What other things can block crawlers?

Robots.txt is only one layer. Firewalls, content delivery networks, security plugins and bot-protection services may block automated traffic regardless of your file, sometimes by default. Owners are often surprised to learn that a security setting has been silently refusing legitimate crawlers for months.

Check your hosting and security dashboards for bot rules, and review server logs for the crawlers you expect. If you decide to allow a crawler, make sure nothing else prevents it. If you decide to block, confirm that the block actually works.

Does allowing AI crawlers guarantee a mention?

No. Access is a precondition, not a promise. Assistants choose what to retrieve and cite based on their own systems, and the page still needs to be clear, relevant and trustworthy. Being blocked can only make things harder, but being allowed does not make you chosen.

Pair your access policy with the other fundamentals: readable pages, direct answers, structured data and consistent business information. An optional llms.txt file may add a helpful guide, though not every system reads it.

Step by step

1. Read your current file. Open yoursite.com/robots.txt and note existing rules and any blanket blocks.
2. Check provider documentation. Confirm the current crawler names and purposes for each AI company.
3. Decide what to allow. Choose which sections to open or close and write down the reasoning.
4. Edit and test. Update the file, then confirm key pages remain reachable by search crawlers.
5. Review security layers. Check firewalls and bot settings so they match your decision.

Frequently asked questions

Will blocking AI crawlers affect my Google rankings?

Blocking AI-specific crawlers does not directly affect search crawlers, provided you do not also block the search engines' own user agents. Test carefully.

Does robots.txt remove content already collected?

No. It governs future crawling by crawlers that respect it. Removal of earlier material is a separate matter handled through each provider's processes.

Do all AI companies obey robots.txt?

Major providers state that they do, but compliance by smaller or unknown bots varies. Use proper security for anything private.

Can I block only part of my site?

Yes. Rules can target specific paths, allowing public pages while disallowing carts, portals or staging areas.

How often should I review robots.txt?

At least a few times a year and whenever you change platforms, security tools or hosting.

Need help with this? See our GEO — Generative Engine Optimisation service or talk to Yash Parikh.