A test named testalladapters_importable asserted nothing. It would pass forever, even if every adapter was broken.
57 of 65 assertion files were in the wrong format, and the harness returned 0 / 0 without a whisper.
Three Times a Green Suite Hid a Real Failure The test gutted to pass. In planner-critic-engine, a test literally had a pass body:
It was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236. The harness that returned 0 / 0 and called it green. In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned 0 / 0 — which it treated as success. The suite was green because it had executed nothing. The auth guard that never ran. In CauterRule v0.3.0 the suite reported 1,558 tests green. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a Docker field test running the real thing in a real container.
In all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the confidence.
We treat "tests pass" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.
The fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.
Two rules fall out of this: CI must fail when a module produces zero results. 0 / 0 is an error state, not a pass. Distinguish "did not run" from "ran and passed." A swallowed import and a skipped module should be loud failures, not silence.
That single distinction is the difference between a flaky gate and a strict one.
Meta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of "green but broken." It does not eliminate it.
But it does close the worst category: the test that never ran and told you everything was fine.
What's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.
Repos and receipts: CauterRule · CauterRule v0.3.0 field test report · PlannerCritic failure-mode register · agent-tooltrust design decisions — all MIT, all public.