Benchmark · LongMemEval-M · full-haystack
77.1% on the variant nobody cites.
Everyone cites LongMemEval. Almost nobody runs the full-haystack: ~480 distractor sessions per question, the honest variant and the hardest one. 77.1% under a strict judge (37/48, reproduced twice, confirmed on a 48-question holdout with zero retrieval regressions), 81.3% under the flexible judge. The gain: a second, fully local retrieval channel (BM25 + rank fusion). The story · the audit section.
July's floor, 72.9% under the flexible judge, stays fully documented below, DOI-pinned as published: the protocol, the failures, and everything you need to audit the score yourself.
What we publish, and what we withhold
LongMemEval-M, full-haystack variant: ~480 distractor sessions per question. Two different judges, so the two figures stand side by side and are never chained into a progression.
Strict judge
77.1%
37/48, reproduced twice, verdict for verdict
Flexible judge
81.3%
The same answers, re-read by July's judge
LongMemEval-S
—
The easy variant the industry publishes on. We publish the full haystack
Both figures recompute from published per-question rows. Recompute them yourself ↗
LongMemEval-S is the easy variant the industry publishes on: 48 sessions of haystack per question instead of ~480. We did run it, in August, on a build that predates the lexical channel and in a single pass. That does not meet the rule the two figures above are held to, a strict judge and a reproduced run, so it stays blank rather than becoming a number nobody can hold us to.
The engine over time
Every number, in order, each with its proof.
Jul 13, 2026
64.6%
The baseline
Retrieval only, no consolidation engine. 31/48 under the flexible judge. The starting line, every verdict published.
The campaign ↓Jul 14–23, 2026
72.9%
The consolidation engine
Multi-session recall 1/8 → 5/8. Published as a composed floor, not a ceiling, and DOI-pinned as it shipped.
The campaign ↓Aug 4, 2026
−31 pts
The lever we refused
Steering retrieval by personality measurably degrades recall. The most seductive feature of the year shipped as a costume, not a driver.
The story →Aug 17, 2026
77.1%
The lexical channel, under a strict judge
A second, fully local ranking, fused by rank. 37/48 strict, reproduced twice, 48-question holdout, zero retrieval regressions. 81.3% under July's flexible judge.
The story →Next
—
A second benchmark family
One benchmark is one benchmark. The reservation stays printed here until a second family is measured.
Every tile's number recomputes from published per-question rows. audit them yourself ↗.
Speed
What it costs to write, and to remember.
Measured on one machine, an Intel Core Ultra 7 155H, CPU only, no GPU, core-engine 1.2.0, on 2026-08-30. Ran twice; the slower run is the one shown. These are our own measurements, not a judged benchmark like the scores above.
Write
0.10ms
p50 into the vault. The embedding that precedes it is not counted
Vector recall
20ms
p50, top 10, warm cache, lexical channel off, 10,200 memories
Hybrid recall
108ms
What the chat actually runs: BM25 fused into the ranking
The hybrid recall is the default since August 17 and is the path behind the 77.1%: it ranks 200 deep and fuses a BM25 pass over the same rows, so it costs 5.3× a vector-only recall. Both answer from the warm in-RAM cache, which is what the app serves. While that cache is still warming, a vector recall falls back to a full scan and costs 145 ms at this corpus size, and that fallback grows with the corpus.
The number, from the July campaign
~480 distractor sessions per question.
The honest variant. The hardest.
There is a more comfortable version of this benchmark. It's the one the industry shows. We ran the other one, and we publish the number as a floor: two categories were never replayed with the full engine, for lack of compute time. Other models still remain to be tested.
No question was picked after the fact. One single configuration for everyone. And a success only counts if it reproduces.
What was measured
| Measure | Before | After | Δ | Conditions |
|---|---|---|---|---|
| Full-haystack, 48 q | 64.6% | 72.9% | +8.3 | all engines |
| Multi-session | 1/8 | 5/8 | +4 | the hardest category |
| Real in-app query path, 61 q | 78.7% | 83.6% | +4.9 | topK 32 |
| Temporal (A/B inline) | 7/10 | 10/10 | +3 | reproduced ×2 |
| Recall at ~1 M vectors | — | 1.000 | — | MNEMO_ANN |
| 100% local, zero cloud | — | ~50% | floor | Qwen2.5-3B · Vulkan · n=12 · loaded machine |
The "100% local" line deserves its precision: it's a floor, not a ceiling. A three-billion-parameter model, on twelve questions, on a machine busy with something else. The big local reasoning models: we do not yet have the power to test them, so we don't know how high it goes, and we don't claim it.
And the levers that gave nothing, because they count just as much: relationship ledgers (worse), master pass (worse), k-means pre-sorting (useless), reserve sweep (noise). Prime numbers carry no memory signal. Falsified six times, buried.
The section nobody writes
What we miss.
On the full-haystack, three questions still resist. They are broken down line by line. Here they are.
×2 · structural to the benchmark
The benchmark stacks several users into a single memory. A stranger's project becomes indistinguishable from yours in the served text. No answer-side filter can decide, verified. Mnemosyne OS is sovereign and single-human: this failure mode doesn't exist for you. We won't chase those two points.
×1 · our debt, not our design
An imbalance between what we write and what we read back: a corpus ceiling truncates a topic that's too dense. It's a real engineering limit, it has a ticket, it will be fixed. We don't dress it up as a choice.
Don't take our word for it
The grader and the verdicts are public. Audit them.
The published grader and the per-question verdicts let you re-derive the score in one command. No engine, no network. The full methodology, the root-cause analysis and the 16 raw logs are open too.