There are two ways to run LongMemEval. There’s the comfortable variant, where the haystack around the right answer is shallow. And there’s the full-haystack: roughly 480 distractor sessions per question, about 2,330 chunks of text in which exactly one fact matters.

Guess which one shows up in the announcements.

We ran the other one. The result is 72.9%, and we publish it as a floor, not a ceiling.

The number, and what it covers

Measurement	Before	After	Δ
Full-haystack, 48 questions, all engines	64.6%	72.9%	+8.3
Multi-session (the hardest category)	1/8	5/8	+4
Real `executeQuery` path, 61 questions	78.7%	83.6%	+4.9
Temporal, inline A/B, reproduced twice	7/10	10/10	+3

Why “floor”: the knowledge-update and single-session categories were never re-run with the full consolidation engine. For want of compute, not of will. The whole campaign cost about twenty hours of compute over two days; re-running those two categories would cost as much again. So the honest number is the one that undersells.

For scale, without comparing products: the reference paper measures GPT-4o with the full context at around 60% — on the easier variant.

The protocol, because the number is worthless without it

Three rules, held across the whole campaign.

The writers are blind. Every consolidation prompt says “consolidate this topic”. Never “answer this question”. The benchmark’s questions are never seen at the moment the memory is written. That’s the difference between measuring a memory and measuring its ability to cheat.

A hit only counts if it reproduces. Every hit was replayed twice before being counted. On one run with six raw successes, two turned out to be judge noise. They were pulled. That’s how you go from a shippable 6/8 to a real 5/8.

One configuration for everyone. No cherry-picked questions, no per-category tuning. The config that yields 72.9% is the same across all 48.

What worked, and why

The gain comes from exactly one place: write time.

While the machine idles, the engine re-reads its own memory topic by topic and writes a ledger — every instance, every value, every date, the running total. That ledger sits beside the raw text, never in its place.

Two details prove that’s where it happens:

The same run with a weaker model as the writer loses the hit. The reader didn’t change. So it wasn’t reading that carried the gain.

And a 40-topic cap on the ledgers killed everything: it kept the big distractor topics and threw away the chunk holding the answer — eight lines lost in the noise. No answer-side filter could recover that. The fact was simply gone.

What we refuted

These measurements don’t usually make the announcements. They should.

Relation ledgers — plausible, tested, worse.
Master pass — plausible, tested, worse.
k-means pre-sort before consolidation — muddy topics, no miss rescued.
Reserve sweep — noise, not a gain.
Prime numbers — measured six ways. No memory signal. Buried.

A regularity falls out of this, and it’s worth more than the points: both real levers of the campaign came from a missing fact traced to its exact position inside its chunk. Both refuted levers came from plausible reasoning with no trace. Since then we don’t build before we trace.

The three we miss

Here’s the part nobody writes.

Two are structural to the benchmark. LongMemEval stacks several users into one memory. A stranger’s project becomes indistinguishable from yours in the served text — we checked: the verification pass keeps them, because they are in the sources. No answer-side filter can adjudicate.

But Mnemosyne is sovereign and single-human. In a real vault there is one person. That failure mode does not exist. We’re not chasing those two points: winning them would mean optimising for a benchmark artefact.

The third is our debt. An imbalance between what we write and what we re-read: a corpus cap truncates a topic that grew too dense. That’s a real engineering limit. It has a ticket. It will be fixed. We don’t dress it up as a design choice.

What we don’t claim

The same document that holds these numbers also holds the list of what isn’t proven, and we keep it current:

Dream bridges do get created, but nobody has measured whether they’re relevant or noise. So we don’t sell them.

Topic-sort quality is unmeasured. The memory organises itself; we don’t claim it organises correctly.

And 359 green unit tests prove that functions work. Not that they’re worth anything. Green units are not proven value.

The convenient pitch would be “we’ll reach 100%, the rest is by design”. That’s wrong by a third, and that third is our work to do.

What we can say fits on one line: 72.9% on the hardest ground there is, published as a floor, with the protocol and the failures printed next to it.

72.9% — and the three questions we miss