Hey everyone. This time I'll go through what got me started on this benchmark, and the core of how it's actually built.
It started from reading complaints, not from an idea. The same ones kept coming up: numbers a vendor publishes don't match numbers someone else measures, swapping the model that does the grading moves the results more than the gap between the systems being compared, and because of that nobody really uses published numbers anyway. They test two options on their own data and keep whichever annoys them less. I work at Wontopos, and we sell a memory API, so I can't point at that and shrug.
All numbers below are from the current build. Nothing is final yet, so some of them will have moved by the time you read this.
I know how "I built a fair benchmark" sounds coming from someone at a memory company. So instead of promising anything, here's what's written in the repo.
Wontopos hosts it and keeps it running. That is the whole role. We also build memory infrastructure, which means we compete in the thing we administer, so the limits are written down rather than promised: Our submissions go through the same approval as everyone else's. We do not merge our own. We do not decide who is admitted. The rules do. Our numbers are verified the same way as everyone else's. Wontopos publishes nothing on a new version for fourteen days. If other memory companies want to co-administer, that's better than us alone, and the offer is open.
A few of the submission rules point the same way: Publish the per-question record. Anyone can recompute the number from it. The aggregate is a claim, the record is the evidence. The reader, the judge and the prompts are set by the version. A submitter does not choose them. The harness has to be one a customer could use. A number produced through a path only its author can reach is not a number anyone else can get.
Everything there is Apache 2.0, so if we ever become the problem, the whole thing can be taken and run elsewhere without asking us.
It's all here: github.com/wontopos/glasshouse. Fair warning about what you'll find. The benchmark itself isn't in there yet and submissions/ is empty. The rules went up first, and we haven't submitted either.
One person's life, told across about 17 months of conversation, with the facts you're supposed to remember buried inside it. 103,572 turns, 1,991 sessions, roughly 1.9M tokens.
It runs at four haystack sizes, from a small core up to the whole thing. The 1,882 turns that actually contain the answers are identical in all four, character for character. What changes is how much unrelated conversation is packed around them, so you can watch a system degrade as the haystack grows instead of getting one score and no idea what it means.
1,547 questions across 14 axes at the full size. Most are ordinary recall: it was said, can you get it back, can you get it when the question uses different words, can you say when it happened, can you combine two facts.
Stale facts. A value changed. The new one exists, but I made it hard to find on purpose. Three outcomes instead of two:
| Answer | Score | |---|---| | New value | 1.0 | | "I don't know" | 0.5 | | Old value, stated as current | 0.0 |
The reason for the middle row is the whole point. Put "I don't know" and "confidently out of date" in the same bucket and you've hidden the thing that actually hurts in production, because those are very different things to be paying for.
Contradictions. The conversation states two different values for the same fact, nobody corrects it, and nothing tells you which is right. There is no correct answer. Confidently picking one is wrong. Saying "these don't match" is right.
Apparent contradictions. The mirror image. Two statements look like they clash but hold under different conditions, so both are true. Telling this apart from a real contradiction is the point.
Abstaining. Questions about things never mentioned at all. An empty answer scores full marks, anything invented scores zero. On top of that there are 500 false-memory probes, which plant something that was never said inside the question itself and check whether the system plays along.
50 photos are shared inside the conversation, with 150 questions about them. Every one of those questions is pinned to a date: "In the photo from 14 April 2026, what was on the sofa?"