We build LovedByAI, a generative engine optimization platform for small business WordPress sites, which means we run crawler-access checks on live sites all day. In August we pointed that at 314 of them, fetching each one as 38 different AI crawlers and comparing the result against four weeks of their server logs.
One finding is a genuine observability trap, and I have not seen it written up anywhere.
A site blocking OAI-SearchBot at the origin logged a page request from it 84.6% of the time. A site with no block at all logged one 87.3% of the time.
Three points apart. So if you are checking your own logs to find out whether AI crawlers can reach you, a working block and no block look the same.
Reading robots.txt does not help either, and that turns out to be the more common failure: of those 314 sites, 135 refused at least one AI crawler and only 23 had written a rule about it.
That is the gap, as our own crawlability view reports it: what robots.txt declares on one side, what the server actually answered on the other. A site can carry one of each at the same time, and only one of them is something a human ever typed.
A block at the edge happens before the request reaches your server. A block at the origin happens after. That single distinction decides everything about what you can observe.
Edge. Cloudflare, a CDN firewall rule, a WAF. The crawler's request is terminated at the network boundary. Your application never runs. Your access log never gets a line. The crawler gets a 403 and leaves.
Origin. A security plugin, an nginx map on $httpuseragent, an Apache RewriteCond. The request completes a TCP handshake, hits your server, gets logged, then gets refused.
So the origin-blocked request appears in your logs as a visit. Your crawler-stats plugin counts it. Your dashboard shows GPTBot stopping by.
Here's the same comparison for the edge case: OAI-SearchBot blocked at the origin: logged on 84.6% of sites OAI-SearchBot not blocked at all: logged on 87.3% of sites OAI-SearchBot blocked at the edge: logged on 0.0% of sites
The edge column is the only one that's legible. Zero requests, because nothing ever arrived. An edge block is loud in your logs by being completely silent. An origin block hides in plain sight.
Of the 135 sites refusing a crawler, only 23 declared it. The other 112 refused at the server with nothing in the site's own config to show for it. The request arrives, the server says 403, and robots.txt still reads like a welcome mat.
And a written rule isn't reliable in the other direction either, because robots.txt is a convention, not an enforcement mechanism: 33.3% of sites disallowing GPTBot still logged real GPTBot page requests 17.6% of sites disallowing ClaudeBot still logged ClaudeBot
Neither figure counts robots.txt or sitemap fetches, which aren't ingested. Every request counted is a crawler asking for a page it was told not to take.
So robots.txt tells you what you meant. Your logs tell you what arrived. Neither tells you what the crawler got.
The part people skip is the control. If you fire ten requests at a site from one IP and get ten 403s, you've learned nothing. It could be your IP, a rate limiter, or the site being down. You need an unbranded request from the same address to prove the site is up and that the refusal is attributable to the user-agent string.
That covers eight crawlers. There are 38 tokens worth checking, including consent switches like Google-Extended and Applebot-Extended that are not crawlers at all and can only be evaluated against robots.txt. Keeping that list current as vendors add and rename tokens is the tedious part. LovedByAI automates the check across all 38 with the unbranded control request built in, and names each crawler that gets refused.