72.9%, and the three questions we miss
Note, September 2026. This post is the July campaign, scored by the flexible judge. On 2026-09-28 an erratum corrected the August figures to 35/48 (72.9%) under the strict judge. The current result is a rerun whose protocol was published before it ran: 77.1% (37/48) on questions the engine had never seen, under the official LongMemEval judge (the benchmark page). The reader, the engine and the judges have changed since July, so these numbers do not form a progression. This post stays up, with its protocol and its admitted composition.
There are two ways to run LongMemEval. The comfortable variant puts little hay around the right answer. The full-haystack puts roughly 480 distractor sessions around each question, about 2,330 chunks of text. Exactly one fact in them matters.
The comfortable one is the one that shows up in the announcements.
We ran the other one. The result is 72.9%. We publish it as a floor.
The number, and what it covers
| Measurement | Before | After | Δ |
|---|---|---|---|
| Full-haystack, 48 questions, all engines | 64.6% | 72.9% | +8.3 |
| Multi-session (the hardest category) | 1/8 | 5/8 | +4 |
Real executeQuery path, 61 questions | 78.7% | 83.6% | +4.9 |
| Temporal, inline A/B, reproduced twice | 7/10 | 10/10 | +3 |
Why “floor”: the knowledge-update and single-session categories were never re-run with the full consolidation engine. We ran out of compute. The whole campaign cost about twenty hours of compute over two days. Re-running those two categories would cost as much again. So the published number is an underestimate.
For scale, without comparing products: the reference paper measures GPT-4o with the full context at around 60%, on the easier variant.
The protocol, because the number is worthless without it
We held three rules across the whole campaign.
The writers are blind. Every consolidation prompt says “consolidate this topic”. Never “answer this question”. The benchmark’s questions are never seen at the moment the memory is written. Without this rule we would measure its ability to cheat instead of its memory.
A hit only counts if it reproduces. Every hit was replayed twice before being counted. On one run with six raw successes, two turned out to be judge noise. They were pulled. The 6/8 we could have shown became the 5/8 we count.
One configuration for everyone. We picked no questions after the fact and tuned nothing per category. The config that yields 72.9% is the same across all 48.
What worked, and why
The gain comes from exactly one place: write time.
While the machine idles, the engine re-reads its own memory topic by topic and writes a ledger. The ledger lists every instance, every value, every date and the running total. It sits beside the raw text. The raw text stays.
Two details prove it:
The same run with a weaker model as the writer loses the hit. The reader did not change. So the gain came from the writing.
And a 40-topic cap on the ledgers killed everything. It kept the big distractor topics and threw away the chunk holding the answer, eight lines lost in the noise. No filter on the answer side could recover that. The fact was gone.
What we refuted
These measurements don’t usually make the announcements. They should.
- Relation ledgers looked plausible. Tested, they score worse.
- A master pass looked plausible. Tested, it scores worse.
- A k-means pre-sort before consolidation gives muddy topics and rescues no miss.
- A reserve sweep produces noise and no gain.
- Prime numbers, measured six ways, show no memory signal. Buried.
One regularity falls out of this, and it is worth more than the points. Both real levers of the campaign came from a missing fact traced to its exact position inside its chunk. Both refuted levers came from plausible reasoning with no trace. Since then we trace first, then build.
The three we miss
Three questions fail. We know why for each one.
Two are structural to the benchmark. LongMemEval stacks several users into one memory. A stranger’s project becomes indistinguishable from yours in the served text. We checked. The verification pass keeps them, because they are in the sources. No filter on the answer side can tell them apart.
But Mnemosyne OS is sovereign and single-human. In a real vault there is one person. That failure mode does not exist. We are not chasing those two points. Winning them would mean optimising for a benchmark artefact.
The third is our debt. A corpus cap truncates a topic that grew too dense, so we write more than we re-read. That is an engineering limit, not a design choice. It has a ticket. It will be fixed.
What we don’t claim
The document that holds these numbers also lists what is not proven. We keep that list current:
Dream bridges do get created, but nobody has measured whether they’re relevant or noise. So we don’t sell them.
Topic-sort quality is unmeasured. The memory organises itself. We do not claim it organises correctly.
And 359 green unit tests prove that the functions work. They do not prove the functions are worth anything.
The convenient pitch would be “we’ll reach 100%, the rest is by design”. That pitch is wrong by a third. That third is our work to do.
What we can say: 72.9% on the hardest ground there is, published as a floor, with the protocol and the failures printed next to it.