I used to just watch. Not read, watch. Scroll the diff as it streamed in, nod at code I hadn't fully parsed, and tell myself I'd review it properly at the end. Sometimes I did. Often I didn't. The question "what do you do while AI codes?" keeps showing up in the dev.to feed, and the comments make it clear I'm not alone. The struggle to stop mid-task and just wait is real.
So I tried to fill the wait with something useful. I built a system where a second model tries to break the first model's work. Then I learned the thing that changed how I think about AI review: 89% of what I built was theater.
I pointed two LLMs at the same pull request and asked them to argue. It worked beautifully: confident claims, point-by-point rebuttals, a clean verdict at the end.
The second model wasn't analyzing anything. It was replaying pre-generated text. Both models were exchanging messages, no positions were changing, no new evidence was being cited. The transcript sounded like a debate. It was a recording of a debate.
If I hadn't looked at the raw output, I would have shipped it and called it a working system. That's the part that still bothers me.
Here's what most multi-model review workflows actually do: Model A reviews the artifact. Model B reviews the artifact plus Model A's output. You call the result independent review.
It isn't. It's a validation. The moment Model B's context contains Model A's verdict, B is anchored. It's doing social-pressure resistance, not analysis. Humans fail this constantly. That's why double-blind review exists. We invented that safeguard decades ago and keep leaving it out of AI pipelines.
The fix is mechanical: Model B cannot see Model A's output until B has fully committed its own position. Not prompted into independence. Structurally prevented from anchoring.
Two models analyze in parallel with zero shared context. Each commits a verdict with structured claims and evidence. Then a bounded debate opens: every objection must reference a specific counter-claim, every claim must cite specific text from the artifact. When they can't resolve, the output is a structured disagreement report, both positions preserved and unresolved points documented.
That last part matters. Most multi-agent systems are engineered toward consensus. Disagreement is treated as a failure to be smoothed over. But if two independent reviewers reach different conclusions with different evidence, that tension is the signal. Collapsing it into one confident verdict throws away the exact thing you built two reviewers to find.
I ran the first real field test on 70 actual pull requests from public repos, 411 debates. The core question: do the claims correspond to what actually went wrong in those PRs? 81% of debate claims matched documented PR outcomes. When it found an issue, it was usually the issue. But I fixed 13 bugs before I trusted any of that: CSV parsers breaking on commas in PR descriptions, model slugs silently routing to the wrong provider, a join bug that collapsed the dataset from 2,333 rows to 359. None were architecture bugs. All were "you haven't run this on real repositories at scale" bugs.
By v0.2.0: 150 artifacts across four domains, 88.7% binary match against the corrected 2,333-row dataset, near-zero fabrications. Total cost: $0.42 for 360 reviewer runs.
Less than lunch. Compute was never the bottleneck. Prompt engineering and ground-truth measurement were.
The best pairing wasn't the two smartest models. It was GPT-4o-mini + Mistral Small 3.2, a small OpenAI model and a small European one.
GPT + Gemini produced the worst debates: lots of rounds, almost no concessions, no resolution. Two RLHF-heavy models from big US labs agree fast because they were trained to prefer harmony. GPT-mini + Mistral produced genuine disputes, real concessions, and eventual convergence.
My working theory: it's not raw capability that predicts debate quality, it's diversity of training objective. A later run refined this: the effect may be as much about non-Mistral models rubber-stamping each other, so the practical rule became "always include Mistral." I hold the theory loosely.
I have not proven this rigorously. It matches everything in my data, and it's why my model-selection advice starts with "pick from different labs" before anything about benchmark scores.
88.7% sounds good. It is good, compared to single-pass review. But there's a problem: