Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
For two field tests, one number refused to move: golden recall sat at 0.087. The reports both named the same culprit — "replay and matcher calibration is now the highest-value engineering target." So we did the obvious thing and worked on the matcher.
Replay grades a candidate trigger against reference trajectories. Recall is the fraction of failures the trigger catches:
A v0.2.0 candidate that prevented 3 real failures — a genuinely useful rule — was scored 3 / ~200 = 0.015. No threshold can admit 0.015 without also admitting noise. The rule wasn't weak. The denominator was the entire corpus.
Ask the question the metric was actually answering: "does this git rule also prevent docker, python, terraform, and browser failures?" Of course it doesn't. It was never supposed to.
Assumption: every failure in the reference pool is a fair test case for every candidate rule.
That assumption is wrong whenever candidates are domain-specific — which, for a real rule engine, is always. The consequence is subtle and dangerous: the metric doesn't fail loudly. It just makes good rules look worthless, uniformly, across every model. And a metric that makes everything look equally bad is a metric that hides where the actual problem is.
We also assumed recall was a model measurement. It is a ratio, and we were controlling the denominator as if it were irrelevant.
The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:
| Candidate domain | Scoped references | Global pool (old) | |---|---:|---:| | git | 19 | ~200 | | python | 30 | ~200 | | docker | 30 | ~200 |
A rule that prevents 3 git failures is now scored 3 / ~27 ≈ 0.11, not 3 / 200 = 0.015.
We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5. That is not a matcher fix. It is the acknowledgement that once scores are computed honestly, 0.8 was calibrated to an inflated scale. Report both or you are hiding the trick.
| Corpus | Model | Recall pre | Recall post | Delta | |---|---|---:|---:|---| | golden | gpt-4o-mini | 0.068 | 0.170 | 2.5× | | golden | llama-3.1-8b | 0.104 | 0.228 | 2.2× | | failures/positive | gpt-4o-mini | 0.068 | 0.182 | 2.7× | | failures/positive | llama-3.1-8b | 0.104 | 0.277 | 2.7× |
No model change. No prompt change. No change to the core matcher scoring. Golden pass rate moved to 40% (gpt-4o-mini) and 50% (llama-3.1-8b).
What worked Recall became diagnosable. Against a global pool, everything looked equally hopeless. Against a scoped pool, the recall signal separates a specific rule from a vague one. The fix is model-independent. It held across both models — the signature of an evaluation bug, not a capability ceiling. It exposed the real bottleneck. Once the denominator was honest, the remaining failures pointed at the matcher's inability to read paraphrases — a concrete, addressable problem.
What didn't work Recall is still far below target. 0.170–0.228 against a 0.70-ish ambition. Doubling a small number leaves a small number. The paraphrase gap is untouched. Semantic matching runs at a 0.2 blend weight and cannot bridge "non-fast-forward" against "Updates were rejected because the remote contains work that you do not have locally." Same event, different tokens. Scoping could flatter reference-rich domains. python and docker have ~30 references; git has 19. We have not measured whether thin domains are systematically penalized. Changing two things at once blurs attribution. Scoping and the threshold drop landed together; the 2–3× is scoping's, but the pass-rate movement is both.
Questions we still can't answer Would a 0.4–0.5 semantic weight close golden, or does the token-F1 term still dominate the blend? Is domain the right scope, or should it be failure-class? We chose domain because the label already exists on every trajectory — a convenience assumption. At what reference count per domain does scoping stop helping? We have no curve, only two field tests. If a candidate has no domain label, do we fall back to the global pool, and does that reintroduce the bug?