AI Crawlers: What to Allow, What to Block

Most robots.txt files we audit were written before answer engines existed, and then patched in a hurry when the first AI crawler made the news. The result is usually one of two mistakes. Either the site blocks everything with AI in the name, including the bots that put it in front of buyers, or it blocks nothing because nobody was sure what each bot did.

The decision is simpler than it looks once you separate the bots by job. There are three jobs: collecting content for model training, building a search index that answer engines cite from, and fetching a page because a user just asked about it. You can say yes to one and no to another. This guide lists the bots by provider, checked against each provider's own documentation, and gives you a robots.txt you can defend.

Three jobs, three decisions

Providers publish their bots with different names because the bots do different things. Treating them as one 'AI crawler' is how sites end up invisible in ChatGPT search while still feeding the training set, which is the opposite of what most owners want.

• Training bots collect pages that may be used to train future models. Blocking them is a policy choice about your content. It does not change whether you appear in answers today.
• Search bots build the index that an answer engine cites from. OpenAI says plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. This is the bot that decides your visibility.
• User fetchers retrieve a page because a person asked a question that needs it. Several providers say robots.txt may not apply to these requests because a user initiated them. You cannot reliably block them, and you rarely want to.

The bots, by provider

The OpenAI, Anthropic, Perplexity and Google rows below were checked against each provider's documentation on 13 October 2026. Providers change these pages without notice, so re-check before you rely on a row. The tokens are the strings you put after User-agent: in robots.txt.

What to decide, by business type

The answer depends on what your content is for. A publisher that sells access to its archive has a different interest from a plumber who wants to be the name an assistant suggests. Work through the three jobs with your own situation in mind.

A robots.txt that says what you mean

Write one group per bot, in the order of the three jobs, so the next person who opens the file can see the policy without a meeting. These five steps produce a file that matches the default we recommend for a business that wants to be found.

1. 01

   Keep the general rules first

   Your existing User-agent: * group, with the admin paths, parameter traps and staging folders you already disallow, stays at the top. AI bots that match no specific group fall back to it.

2. 02

   Allow the search bots explicitly

   Add a group each for OAI-SearchBot, Claude-SearchBot and PerplexityBot with Allow: / and the same sensitive-path disallows as the general group. Explicit groups stop a later edit to the wildcard from silently blocking them.

3. 03

   Decide the training bots

   Add groups for GPTBot, ClaudeBot and CCBot. Either Allow: /, or Disallow: /, or a path list if you want docs allowed and the archive blocked. Add Google-Extended and Applebot-Extended here too; they are tokens that only take effect in this file.

4. 04

   Leave the user fetchers alone

   Do not add groups for ChatGPT-User, Claude-User or Perplexity-User. Providers say these requests may ignore the file, and a block that half works only produces confusing logs.

5. 05

   Test it

   Fetch your own pages with curl -A "OAI-SearchBot/1.0" and the other tokens, and check that the server returns 200 and the full HTML, not a challenge page. Then check robots.txt itself is reachable without a challenge, because a bot that cannot read the file cannot obey it.

The mistakes we keep finding

• Blocking by keyword. A rule that matches anything containing 'GPT' or 'Claude' catches the search bots along with the training bots. Match the exact tokens.
• Blocking at the edge and forgetting. A WAF rule added during a scraping scare in 2024 is still challenging OAI-SearchBot in 2026. Nobody sees it, because nothing on screen looks wrong.
• Confusing Disallow with noindex. Disallow stops crawling. It does not remove a page that is already known, and a disallowed page cannot show the noindex tag that would remove it. If you want a page out of answers, let it be crawled and mark it noindex.
• Blocking robots.txt itself. Challenge pages on /robots.txt mean every bot assumes it is allowed, or gives up, depending on the bot. Either way your policy is not in force.
• Shipping the staging file. A staging robots.txt with Disallow: / for everyone goes live with a release and takes the search bots with it. Diff the file on every deployment.
• Treating Google-Extended as a crawler. It never appears in logs. People search for it, do not find it, and conclude Gemini is not visiting. Googlebot is doing the visiting; Google-Extended only sets what the crawl may be used for.

How to check what the bots actually see

Policy is one thing; what a bot receives is another. Three checks tell you the truth, and together they take under an hour.

1. Logs. Filter your server or CDN logs for the tokens above over the last thirty days. Count requests and status codes per bot. A search bot that only ever sees 403 or 429 is blocked, whatever robots.txt says. No requests at all usually means the bot has never found a link to you, which is an authority problem, not an access problem.
2. Fetch as the bot. Request your home page, a category page and an article with each token as the user agent. Compare the HTML to what a browser gets. If the main content only appears after JavaScript runs, the bot is reading a shell, and that is a rendering fault to fix before anything else.
3. Ask the engines. Run the questions from your prompt panel and note which pages are cited. If the answer cites a competitor's version of a page you also have, the problem is usually not access but clarity, and the fix is on the page.

The bottom line

• Separate the three jobs. Search bots decide your visibility; training bots are a policy choice; user fetchers are not reliably blockable.
• Write explicit robots.txt groups for the search bots, decide the training bots and the control tokens deliberately, and leave the user fetchers alone.
• Verify with logs and a fetch as each bot. Most invisibility we find is a firewall rule, not a robots.txt line.

Frequently asked questions

If I block GPTBot, will my site disappear from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot are the ones not shown in search answers. You can block training and still be cited.

Does Google-Extended control AI Overviews?

Google's documentation says Google-Extended controls whether content may be used to train future Gemini models or to ground Gemini answers, and that it does not impact inclusion in Google Search. AI features inside Search are served from the Search index that Googlebot builds, so Googlebot access is what matters there.

Can I block the user-triggered fetchers?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its bots honour the file. In practice, blocking these only makes a reader's assistant fail on your page.

How often should I re-check this?

Read the provider pages once a quarter and after any news about a new bot. Check your own logs monthly. The provider pages change without notice, and a CDN update can change bot handling on its own.

How this guide was put together

• The OpenAI, Anthropic, Perplexity and Google rows were read from each provider's documentation on 13 October 2026. The remaining rows are from those organisations' published documentation and were not re-read that day.
• Recommendations reflect what we do on client sites. Where a provider's wording leaves room for doubt, the table quotes the provider rather than our interpretation.

Sources

1. OpenAI: Overview of OpenAI crawlers
2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
3. Perplexity: Perplexity crawlers
4. Google Search Central: Google's common crawlers, including Google-Extended
5. Google Search Central: Robots meta tag and X-Robots-Tag
6. Lucidens: How to Get Cited by AI Answer Engines
7. Lucidens: How to Run a Prompt Panel for AI Citations