There's a lot of guessing about whether AI systems are citing your site. Analytics won't tell you — most of these referrals arrive with no referrer at all.
Your access log will, and it's more specific than people realize: two different classes of user-agent mean two completely different things, and only one of them is evidence that you were actually cited.
I've been logging this on a small site since launch. Here's the mechanism, the distinction that makes it useful, and the trap that will make you over-count.
Bucket 1 — crawlers. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, Applebot-Extended, CCBot. These are building or refreshing an index. A hit here means you're in a crawl queue. That's necessary, it's not an outcome, and it says nothing about whether anyone ever saw your content in an answer.
Bucket 2 — user-triggered fetches. ChatGPT-User, Perplexity-User, Claude-User. Different job entirely: a person was in a conversation, and the assistant fetched your page to answer them — either browsing live or following a citation the user clicked.
That second bucket is the closest thing to direct evidence that your page participated in an answer. Naming convention across vendors is conveniently consistent: -Bot and -SearchBot suffixes are infrastructure, -User suffix means a human is on the other end.
Most write-ups on this topic lump all of these into one "AI traffic" number, which throws away the only distinction you actually care about.
My stack has no nginx in front of the app, so there's no access log on disk to grep. If you're in the same position, you don't need to add one — the edge proxy can log the hit itself.
Archive out of the log buffer. Process managers rotate, restart, and flush. A -User hit may appear exactly once, ever — if it lands in a buffer that gets cleared, it's gone and you'll never know it happened. A small script on a periodic cron that appends new lines into an append-only file, deduped, is enough. Make it idempotent so re-running is free.
Check what your matcher excludes. Mine initially skipped llms.txt, robots.txt, and sitemap.xml, which are precisely the files an AI crawler hits first. Those counts read as zero for two days and the zero was an artifact. If your proxy has a path matcher, audit it against the paths bots actually want.
Here's the part I hadn't seen written up anywhere, and it will corrupt your data if you don't handle it.
Some of your "AI crawler" traffic is a vulnerability scanner wearing a costume. One 83-second burst on my site: 60 requests rotating through 6 different AI user-agent strings, hitting paths like /aws-credentials and /id_rsa, plus query strings carrying shell-injection probes.
Spoofing a user-agent is trivial, and AI crawler strings are a good disguise precisely because everyone has decided to allow them.
If you count those as AI interest, you'll conclude the robots love you. Tag them instead:
Two deliberate choices there. Blacklist suspicious paths rather than whitelisting your real ones — if a blacklist misses something you get a few junk lines, but if a whitelist misses a page you just shipped, it misclassifies a genuine citation as a scan. One failure mode is noise; the other is silently destroying your only copy of a signal that occurs once.
And keep the scan lines, just tagged differently. They're still useful for security review; they just don't belong in the citation count.
I wrote the section above from a single 83-second burst. Then I left the log running and came back on 2026-09-12 — 3,392 lines, five and a half weeks. The naive count:
| User-agent | Raw hits | |---|---| | ChatGPT-User | 1,462 | | ClaudeBot | 872 | | OAI-SearchBot | 443 | | PerplexityBot | 385 | | GPTBot | 177 | | Claude-User | 19 | | Claude-SearchBot | 10 | | Perplexity-User | 6 |