"Does your AI code reviewer read CLAUDE.md?" is the wrong question to ask. Almost any of them will say yes, and honestly, most will catch something. What actually matters is whether you can tell the difference between a tool that read your rule and a tool that just happens to agree with it.
So I built a small API, wrote real rules into CLAUDE.md and AGENTS.md, and then broke one of those rules twice on purpose. The first break was something any decent security scanner would flag anyway. The second one had no reason to get caught unless a tool actually opened the file and read it. I ran both through CodeRabbit and Qodo to see what would happen.
Both tools caught the second one eventually. Only one of them could tell me why.
rules-demo-api is a small Cloudflare Worker. POST /feedback takes a rating and a message, validates them, and logs the rating. Both CLAUDE.md and AGENTS.md document the same four rules for the repo, including this one:
"Never log request headers or bodies. They may contain secrets, tokens, or PII."
"All new route paths must be kebab-case (/feedback-summary), never camelCase (/feedbackSummary). This is an internal convention, not a general best practice."
I wrote that second rule that way on purpose. There's nothing about kebab-case routes that's a security issue, and nothing about it is a bug either — it's just a house style choice with zero backing outside this one repo. Which is exactly what makes it a cleaner test than the first rule. If a tool flags a violation of this one, it had to have read the file, because there's nothing else it could be catching. It's a small example, but this is exactly the kind of gap that shows up once you're running AI code review at scale instead of just eyeballing one repo yourself.
I opened a PR that logs the full request headers and body, with a commit message saying it was "temporary, to debug a client issue." Both tools caught it right away.
Rule violation, High: Customer secrets enter application logs "Rule 1: Do Not Log Unredacted Request Headers or Bodies" (AGENTS.md[5-8])
"This change can expose credentials and private feedback content through application logs. It should not merge until logging is restricted to an allowlist of necessary diagnostic fields." Merge Risk: High.
Both correct, and only one of them names the actual file. But this test doesn't really prove anything about file-reading either way: logging raw headers and bodies is a well-known bad practice whether or not you've written it down anywhere. Any half-decent scanner flags this on instinct alone. Call this round a wash.
Same repo, new PR. This time I added a GET /feedbackSummary endpoint in camelCase, breaking the kebab-case rule and nothing else. No security issue, no bug, just a naming convention nobody outside this repo has ever heard of. The only way to catch it is to have actually read CLAUDE.md or AGENTS.md.
Rule violation, High: Clients cannot use the kebab-case route "Rule 4: Use Kebab-Case for New API Route Paths" (AGENTS.md[7-10], CLAUDE.md[7-10]) "Requests to /feedback-summary... do not match the condition and fall through to the handler's 404 response."
It even took the reasoning a step further than I expected. Since the handler 404s on anything unmatched, Qodo worked out that a client calling the correct kebab-case URL would get a 404, while my wrong, camelCase URL worked fine. That's not just matching my code against a style guide. That's tracing what the naming mismatch does once real requests hit it.
"No actionable comments were generated in the recent review. 🎉" Merge Risk: Minimal.
Nothing. A clean bill of health on a PR that breaks a rule written into both files in the same repo CodeRabbit had just finished reviewing.
Before I wrote CodeRabbit off, I went and checked its settings. There's a Chill/Assertive toggle in there, and Assertive is documented to surface more style-level feedback. So I switched it over and forced a full re-review on the same PR.
"Update the route condition... to match the required kebab-case path /feedback-summary instead of /feedbackSummary."