It took one line of curl and a user agent string copied out of Anthropic's own documentation. Three requests to one article page. Then I opened my dashboard: Anthropic, ClaudeBot, training — 1,698 requests had become 1,701, last seen at 09:22.
None of them was ClaudeBot. They came from a laptop in Japan, over ordinary home broadband, from an address that appears on no crawler list anywhere.
The instrument was not broken. It was doing exactly what it is built to do, which is to write down what arrived and what it called itself. A user agent is a claim. Anyone can type it. The whole question is how much of what you are measuring is a claim you can check, and the honest answer is worse than I expected — including, as it turned out, in ways that were my own fault.
There is a way to check, for some of them. A vendor can publish the IP addresses its crawlers run from. If a request claiming to be GPTBot arrives from an address on OpenAI's published list, it is GPTBot. If it arrives from my laptop, it is not. Some vendors publish such a list. Some publish nothing at all. So my dashboard has a column called Verified, and it is the least glamorous number on the page.
Here is one site, davaonline.net, over the thirty days ending September 4, 2026: 13,491 crawler requests, of which 10,390 landed on pages with actual writing on them. Sorted by volume, with the share of each that could be confirmed:
| Crawler (requests) | Verified | | --- | ---: | | meta-externalagent (5,728) | no method published | | ClaudeBot (1,698) | no method published | | Applebot (1,479) | 98% | | Perplexity-User (984) | 0% | | ChatGPT-User (887) | 6% | | Bytespider (767) | no method published | | GPTBot (598) | 81% | | CCBot (440) | no method published | | Amazonbot (342) | no method published | | OAI-SearchBot (242) | 60% | | meta-webindexer (110) | no method published | | PerplexityBot (103) | 0% | | Claude-User (69) | no method published | | GoogleOther (25) | 100% | | Claude-SearchBot (19) | no method published |
## 1 in 6 Requests that arrived with proof of who sent them. Roughly 2,200 confirmed out of 13,491. Everything else is a string I wrote down because it was handed to me.
That single number hides three completely different failures, and separating them is the only useful thing I can do here.
| All 13,491 requests, by why verification did or did not happen | | | | --- | ---: | ---: | | No verification method published by the vendor | 6,947 | 51% | | Published, but my bundled copy was out of date | 2,226 | 16% | | Published, I had it, and the address did not match | 4,318 | 32% |
Meta, ByteDance and Amazon publish no IP ranges and no reverse-DNS scheme for the crawlers in that table. There is no list to compare an address to. Not a hard case, not an unlucky one — there is simply no mechanism, so verification is not slow or partial, it is impossible.
That accounts for 6,947 of the 13,491 requests. More than half the traffic in this table comes from vendors who have decided that site owners do not need a way to tell whether it is really them. The largest single crawler on the site, meta-externalagent at 5,728 requests, is in this group.
Amazon is a near miss worth naming: it does publish a page of addresses for Amazonbot, but as a web page whose contents only appear after JavaScript runs. A list a machine cannot read is a list that does not exist for this purpose.
And a small trap for anyone else doing this: ByteDance's obvious candidate URL returns HTTP 200. The body is a 404 page. If you check for a published list by looking at status codes, you will conclude they publish one.
Look at the Anthropic rows and the CCBot row in that table. They say no method published. That was true when the plugin's bundled snapshot of vendor ranges was taken, on July 20. It is not true now.
Anthropic began publishing a range list on August 18, replacing an explicit statement that it did not. Common Crawl published one on August 11. Both landed inside the thirty days measured above, and my instrument went on reporting the old world, because a design decision I am otherwise happy with — the plugin never talks to the network, so vendor ranges ship as a snapshot — means the snapshot goes stale silently.
That is 2,226 requests, sixteen percent of the total, reported as unverifiable when they were merely unverified by me. I fixed it on September 4; the count of crawlers this instrument can verify went from eleven to fifteen. The fix also caught a Google crawler that had been listed as verifiable for months while its ranges were never actually bundled.
I would rather not have found this while writing an article about verification. But it is the more useful half of the lesson: whether a crawler is verifiable is not a property of the crawler. It is a property of how fresh your copy of somebody else's list is, and it decays without telling you.
Perplexity publishes IP ranges. I had them. Over thirty days, 984 requests arrived calling themselves Perplexity-User and 103 calling themselves PerplexityBot. The share of them coming from an address on Perplexity's own published list was not low.