Sooner or later someone forwards you a screenshot of the access log and asks whether to block "the AI bots". It is a reasonable question with an unreasonable premise: there is no single population there. There are at least three, they want different things, and blocking them costs you different things.
Training crawlers collect pages to train a future model. Nothing you do today changes a model that already shipped, and nothing about your search ranking depends on them.
Live-fetch bots fetch one page because a user asked something right now. Block these and you disappear from the answer that was about to cite you.
Answer-engine indexers build an index used to answer questions with citations. Closest thing to search, and the one where blocking most directly costs you referral traffic.
The same company usually runs one of each, under different names. That is the part people miss.
| User-agent | Operator | Job | What blocking it costs you | |---|---|---|---| | GPTBot | OpenAI | Training | Nothing in the short term | | OAI-SearchBot | OpenAI | Search index for ChatGPT | Visibility in ChatGPT search | | ChatGPT-User | OpenAI | Live fetch on user request | Your page cannot be opened when a user asks about it | | ClaudeBot | Anthropic | Training | Nothing in the short term | | Claude-Web | Anthropic | Live fetch | Same as ChatGPT-User | | PerplexityBot | Perplexity | Index for cited answers | Citations, and the referral traffic with them | | Google-Extended | Google | Gemini training only | Nothing in Search — see below | | CCBot | Common Crawl | Open dataset many models train on | Nothing directly; wide indirect reach | | Bytespider | ByteDance | Training | Nothing in the short term | | Applebot-Extended | Apple | Apple Intelligence training | Nothing in Search — Applebot is separate | | Amazonbot | Amazon | Assistant and search | Alexa-surface visibility | | meta-externalagent | Meta | Training | Nothing in the short term |
It controls whether your content is used to train Gemini. It has no effect on crawling for Search, on indexing, or on ranking. Blocking it does not deindex you and does not cost you a position.
Googlebot is what crawls for Search. If you block that one, you leave the index.
I have seen the two conflated in both directions: people who avoid blocking training because they fear losing rankings they were never going to lose, and — worse — people who block Googlebot believing they are opting out of AI. The names are similar; the consequences are not.
Same pattern at Apple: Applebot crawls for Siri and Spotlight, Applebot-Extended is the training opt-out. Different bots, different consequences.
For scale, we crawled the robots.txt of a large sample of sites for a study on this exact question: 17.8% block GPTBot against 2.5% for Googlebot, and 12.3% block PerplexityBot, a crawler that does not train models at all. That last figure is the accidental-damage bucket, measured in the wild.
The argument for blocking training crawlers is simple: your content is an asset, training on it is a use, and you get nothing back.
The argument against is subtler. Assistants that cite sources send traffic, and increasingly they are the search interface. A site that is invisible to them is invisible in a place where people now ask questions. And the distinction between "training" and "answering" is blurrier in practice than the bot names suggest.
The position most people land on, once it is laid out this way: Block training — GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent. Allow live-fetch and answer indexes — ChatGPT-User, Claude-Web, OAI-SearchBot, PerplexityBot. Never touch Googlebot, Bingbot.
That is a position, not a default. A documentation site that wants to be the answer everywhere might block nothing. A publisher whose archive is the product might block everything. Both are coherent; drifting into one by accident is not.
A specific group replaces the wildcard group — it does not extend it. If you write a group for GPTBot, that group is the whole ruleset for GPTBot. Rules under User-agent: are not inherited. This is the most common way a hand-written file ends up doing the opposite of what was intended.
robots.txt is a request, not a fence. Well-behaved crawlers obey it. Nothing enforces it. If a bot must be stopped rather than asked, that is a job for authentication or blocking at the edge, and the operator list above is the wrong tool.
Test the file before you ship it. The precedence rules are unintuitive: the longest* matching rule wins, not the first, and on a tie Allow beats Disallow. A file that reads correctly to a human can behave differently to a parser.