There is a failure mode that does not show up in any of your dashboards. Your site ranks. Analytics look normal. You open the page in a browser and it is obviously fine. And an assistant, asked about the exact thing you built, has nothing to say about you — not because it judged your product, but because it never received any words.
The cause is unglamorous. Googlebot executes JavaScript. The crawlers behind ChatGPT, Claude and Perplexity do not. Vercel and MERJ measured this across more than 500 million GPTBot fetches and found no JavaScript execution at all. So a single-page app that assembles its content in the browser is two different documents depending on who asks for it: a full page for Google, and whatever sits in the initial HTML for everyone else.
If that initial HTML is a and a script tag, that is the whole site as far as an assistant is concerned.
This is worth being precise about, because "AI can't read my site" gets said in a vague way that leads people to buy the wrong fix.
Nothing is being blocked, hidden or penalised. The crawler makes an ordinary HTTP request and gets an ordinary 200 response. It simply keeps what the server sent and stops there. Google's crawler does a second pass and runs the scripts; the AI crawlers have no second pass.
The practical consequence is that every rendering choice you made for performance or developer convenience now has a distribution consequence you were not thinking about. Content behind a client-side fetch, a tab that loads on click, a pricing table hydrated from an API, a description rendered by a framework with no server pass — all of it is present for a human, present for Google, and absent for an assistant.
The empty shell. The server returns 200 and about forty words: a title, a noscript line, maybe a nav. Everything that explains the product arrives later, in JavaScript. This is the classic client-rendered app, and it is the hardest to notice because nothing anywhere reports an error.
The 403. Bot protection at the CDN or firewall refuses the crawler's user agent before your application is ever reached. Your robots.txt says the crawler is welcome; your edge disagrees, and the edge wins. Nobody chose this — it usually arrives as a default when someone turns on bot filtering.
The accidental robots rule. Someone added a blanket disallow to keep the site out of AI training and took out the search crawler in the same line. Which brings up the distinction almost everybody gets wrong.
GPTBot collects training data. OAI-SearchBot fetches pages for the index behind ChatGPT's search citations. They are independent, and blocking one has no effect on the other.
Block GPTBot and you have opted out of training runs. That is a legitimate choice and it costs you nothing in ChatGPT's answers. Block OAI-SearchBot and you are removed from those answers entirely — you cannot be cited from an index you are not in.
A blanket User-agent: disallow aimed at "AI" does the second thing while people believe they are doing the first. If you have ever pasted an AI-blocking robots snippet from a blog post, this is worth thirty seconds of your time.
When we checked 51 startup directories for dofollow links, our checker fetched a real listing page and read the rel attribute off the outbound link. Most of what we could not verify was bot protection returning 403 — the second shape above, aimed at us.
But two directories defeated the check for the first reason: they render their outbound links client-side. The anchor does not exist in the HTML that the server sends. Our dofollow checker sees no link at all, and neither does anything else that does not run a browser.
Those directories are almost certainly fine for humans. The point is narrower and worth sitting with: if a link only exists after JavaScript runs, then every consumer of your page that does not run JavaScript is working from a document with no link in it. The same logic that hides your product description from an assistant hides an outbound link from a crawler.
You cannot see any of this in a browser, because your browser is the one thing in this story that definitely runs JavaScript. You have to fetch the page the way a crawler does.
If the byte count is small and the grep returns 0, the text is not in the document. Repeat with ClaudeBot and PerplexityBot, because a block can be per-agent, and check the status code separately — curl -o /dev/null -w "%{http_code}" will tell you whether you are looking at an empty shell or a refusal.
We built the AI crawler check to do this without the terminal: it fetches your page as each crawler, once, and shows the status code each one got and the text each one received. It is free and there is nothing to sign up for.