A bug fix can make every new test pass and still introduce a regression. Here is a deliberately constructed Python example, checked locally without an LLM.
An order filter has three requirements: Omit the filter, or pass None: return all orders. Pass an empty list: return no orders. Pass a list of statuses: return only matching orders.
The reported bug is that omitting the filter returns nothing. This proposed fix looks reasonable:
Both checks pass. Both branches of the if have been exercised. The reported symptom is fixed.
Python treats both None and [] as falsey. Our requirements give them different meanings, and the patch erases that distinction.
The connection to coding agents becomes more consequential when those checks determine what the agent does next.
On September 8, Leitian Tao and colleagues published the ExecCritic preprint. Holding the Qwen-3.5-35B-A3B Repair agent fixed, they reported these SWE-bench Verified results:
| Feedback source | Tasks resolved | |---|---:| | Initial repair, before generated-test feedback | 61.2% | | Tests from the base Qwen Test agent | 57.3% | | Tests from GPT-5.6-sol | 65.3% |
The weaker tests reduced the resolved rate by 3.9 percentage points. Better tests improved it.
Rates average three repair runs, reusing generated tests. Failed test qualification retains the initial patch in the all-task score. The baseline does not forbid repository tests. A separate official evaluator determines resolution. Feedback adds test-generation and revision work; compute budgets are not matched. These are the authors' results, not a benchmark replication for this article. Method and results.
A bad test can do more than miss a defect. It can give the next edit the wrong target.
Our broken patch passes it. A correct implementation would fail it. Feed that failure into an automatic repair loop, and the loop now has a reason to damage correct behavior.
Even the familiar “fails before the fix, passes afterward” check needs a closer look. Here is the original implementation from the fixture:
The default-filter assertion fails against this version and passes against our proposed patch. It correctly detects the original bug. It simply cannot detect the new one.
| Implementation | Default + paid-filter checks | Those checks + empty-list check | |---|---|---| | Original | 1 passes, 1 fails | 2 pass, 1 fails | | Plausible patch | 2 pass | 2 pass, 1 fails | | Corrected patch | 2 pass | 3 pass |
The runnable companion includes all three implementations, the checks, and the verified output. It uses Python’s standard library and makes no LLM or network calls.
The extra check earns its place because it distinguishes two implementations the earlier checks considered equally acceptable.
That is the question I would bring to an AI-generated test review: which plausible wrong implementation would this test reject?