I was about to start a monthly column about AI search visibility — how to get your pages cited by ChatGPT, Perplexity, Google's AI Mode. Medium was the obvious venue. Big audience, publications that distribute for you, no infrastructure to run.
Checked 2026-09-17, HTTP 200. Alongside the usual rules there's a second group:
Eight user-agents, Disallow: /, and an allow-list containing exactly the pages that sell Medium. Your post is not on that list.
That's a defensible business decision — Medium licenses its archive and doesn't want it taken for free. It's just the opposite of what I needed from the platform.
This is where most takes on this get sloppy, so it's worth being precise. "AI crawler" covers two jobs, and Medium only closes one of them.
Training and bulk collection — closed. GPTBot (OpenAI), ClaudeBot (Anthropic), Applebot-Extended (Apple Intelligence), meta-externalagent (Meta), Bytespider (ByteDance), Amazonbot.
Search-grounded retrieval — open. These are not in the blocked group, so the rules apply and they're allowed:
| user-agent | what it feeds | |---|---| | Googlebot | Google's index → AI Overviews, AI Mode | | Bingbot | Bing's index → Copilot | | PerplexityBot | Perplexity's own index | | OAI-SearchBot | ChatGPT's search index | | ChatGPT-User | user-triggered fetch when ChatGPT opens a link | | Google-Extended | Gemini / Vertex training use of Google-crawled pages | | CCBot | Common Crawl |
So a Medium post can absolutely turn up as a cited source in Perplexity or an AI Overview. It reaches those engines through a search index, and those indexes are open.
Google-Extended isn't blocked. It doesn't crawl anything itself — it's the token that controls whether content Google already crawled may be used for Gemini training. Left unblocked, that path stays open while OpenAI's and Anthropic's equivalents are shut.
CCBot isn't blocked either. Common Crawl is a public archive that a lot of training corpora are built from. Blocking the labs' own crawlers while leaving the archive they can download open is a narrower measure than it first looks.
I don't read this as a loophole anyone left on purpose. I read it as the normal state of a robots.txt: a list of names, maintained by hand, always slightly behind a user-agent landscape that adds a new one every few months.
| platform | AI crawler rules in the group | |---|---| | medium.com | separate group blocking 8 AI user-agents | | dev.to | none — only //actions_panel disallowed | | hashnode.com | none — Allow: / | | substack.com | none — only /action/ disallowed | | news.ycombinator.com | none — open, Crawl-delay: 30 | | reddit.com | User-agent: + Disallow: / — everything blocked |
Reddit is the interesting one. It doesn't single out AI crawlers; it disallows all of them, then licenses the archive to Google and OpenAI directly. Reddit threads do reach those two models, but through a contract, not through a crawl. If you were counting a Reddit comment as third-party evidence any model could go verify, that's not what it is.
A model reading your own domain treats what it finds as a self-description. The same sentence on a domain you don't control reads differently — and that difference is the entire reason to publish somewhere other than your own site.
I have a concrete case. A comparison prompt to Gemini described my tool as a "Local SEO & AI Indexing" product "built on top of Scrapeless." There is no such product underneath it. The acronym did the damage: a lot of indexed repositories use GEO and "geographic" interchangeably, so a model reading a bare GEO with no nearby context can resolve it to the geographic sense and attach whatever vendor sits in that space.
The fix was boring. Put the expansion next to the name where a model reads first — hero paragraph, meta description, the WebSite and WebApplication nodes in the JSON-LD graph. On the built HTML, the entity-disambiguation check went from 1/3 to 3/3 and the citability score from 60 to 62.
But that fix only works on pages I own. The correction that matters more is the one on a domain that isn't mine — which is why the platform's robots.txt is a prerequisite, not a detail.