Benchmark · LongMemEval-M · full-haystack
77.1% on questions the engine had never seen.
Everyone cites LongMemEval. Almost nobody runs the full-haystack variant, which puts ~480 distractor sessions in front of every question. We ran it under a protocol published before the run. On 48 questions the engine had never answered, the official LongMemEval judge scores 77.1% (37/48). On the 48 questions we used to tune the engine, our strict judge scores 85.4% (41/48). Hybrid mode: embeddings and search run on the machine, and a cloud model, gemini-3.8-flash, writes the answers. A question counts only when two separate passes both get it right. The protocol and every answer.
In September an outside audit found a wrong verdict in our August result. We corrected it and published an erratum, then ran the whole benchmark again. The earlier campaigns stay documented further down, as published.
Since September, a second benchmark sits next to it. On BEAM, graded by its own official judge, we score 61.7% at the 100K tier and 49.2% at 10M. The details.
What we publish, and what we withhold
LongMemEval-M, full-haystack variant: ~480 distractor sessions per question. Two sets of 48 questions and two judges. The figures stand side by side and never chain into a progression.
Unseen questions
77.1%
37/48 on a pre-registered holdout, official LongMemEval judge
Tuning questions
85.4%
41/48 on the questions used to tune the engine, strict judge
LongMemEval-S
—
The easy variant the industry publishes on. We publish on the full haystack
Both figures recompute from the published answers with one command. Recompute them yourself ↗
LongMemEval-S is the easy variant the industry publishes on: 48 sessions of haystack per question instead of ~480. We ran it in August, on an older build, in a single pass. The figures above follow a protocol published before the run, with two passes. That August pass meets neither rule, so the box stays blank rather than carrying a number nobody could hold us to.
The engine over time
Every number, in order, each with its proof.
Jul 13, 2026
64.6%
The baseline
Retrieval alone, with no consolidation engine. 31/48 under the flexible judge. That is the starting line, and every verdict is published.
The campaign ↓Jul 14–23, 2026
72.9%
The consolidation engine
Multi-session recall went from 1/8 to 5/8. We published it as a composed floor, not a ceiling, and pinned it to its DOI as it shipped.
The campaign ↓Aug 4, 2026
−31 pts
The lever we refused
Steering retrieval by personality measurably degrades recall. So the most seductive feature of the year shipped as a costume, not a driver.
The story →Aug 17, 2026
72.9%
The lexical channel, under a strict judge
A second ranking, fully local, fused by rank. 35/48 under the strict judge, after the erratum of September 28: an outside audit found a wrong verdict, and we first published 77.1%.
The story →Sep 13, 2026
49.2%
A second benchmark: BEAM
BEAM (ICLR 2026), graded by its own official judge: 61.7% at the 100K tier, 49.2% at 10M. A 76× larger haystack costs 20% of the score. In the paper, the RAG baselines lose 22 to 29%.
The comparison →Sep 30, 2026
77.1%
The rerun, published before it ran
37/48 on 48 questions the engine had never answered, official LongMemEval judge, two passes. On the 48 tuning questions, 41/48 under the strict judge.
Every answer ↗Every tile's number recomputes from published per-question rows. audit them yourself ↗.
Speed
What it costs to write, and to remember.
We measured on one machine, an Intel Core Ultra 7 155H, CPU only, no GPU, core-engine 1.2.0, on 2026-08-30. We ran it twice and show the slower run. These are our own measurements, not a judged benchmark like the scores above.
Write
0.10ms
p50 into the vault. The embedding before it is not counted
Vector recall
20ms
p50, top 10, warm cache, lexical channel off, 10,200 memories
Hybrid recall
108ms
What the chat actually runs: BM25 fused into the ranking
The hybrid recall has been the default since August 17, and it is the path behind the figures above. It ranks 200 deep, then fuses a BM25 pass over the same rows. That costs 5.3× a vector-only recall. Both answer from the warm in-RAM cache, which is what the app serves. While that cache is still warming, a vector recall falls back to a full scan. At this corpus size the scan costs 145 ms, and it grows with the corpus.
The number, from the July campaign
~480 distractor sessions per question.
The honest variant. The hardest.
There is a more comfortable version of this benchmark. It is the one the industry shows. We ran the other one. We publish the number as a floor, because two categories were never replayed with the full engine, for lack of compute time. Other models still remain to be tested.
No question was picked after the fact. We used one configuration for every question. A success only counts if it reproduces.
What was measured
| Measure | Before | After | Δ | Conditions |
|---|---|---|---|---|
| Full-haystack, 48 q | 64.6% | 72.9% | +8.3 | all engines |
| Multi-session | 1/8 | 5/8 | +4 | the hardest category |
| Real in-app query path, 61 q | 78.7% | 83.6% | +4.9 | topK 32 |
| Temporal (A/B inline) | 7/10 | 10/10 | +3 | reproduced ×2 |
| Recall at ~1 M vectors | — | 1.000 | — | MNEMO_ANN |
| 100% local, zero cloud | — | ~50% | floor | Qwen2.5-3B · Vulkan · n=12 · loaded machine |
The "100% local" line deserves its precision. It is a floor, not a ceiling. We ran a three-billion-parameter model on twelve questions, on a machine busy with something else. We do not yet have the power to test the big local reasoning models. So we do not know how high it goes, and we do not claim it.
The levers that gave nothing count just as much. Relationship ledgers made it worse. So did a master pass. K-means pre-sorting was useless, and the reserve sweep was noise. Prime numbers carry no memory signal. We falsified that lead six times, then buried it.
The failures
What we miss.
On the full-haystack, three questions still resist. Here is each one, broken down line by line.
×2 · structural to the benchmark
The benchmark stacks several users into a single memory. A stranger's project becomes indistinguishable from yours in the served text. We verified that no answer-side filter can tell them apart. Mnemosyne OS is sovereign and serves one human, so this failure mode doesn't exist for you. We won't chase those two points.
×1 · our debt, not our design
A corpus ceiling truncates a topic that is too dense, so what we write and what we read back fall out of balance. It is a real engineering limit. It has a ticket and it will be fixed. We don't dress it up as a choice.
Don't take our word for it
The grader and the verdicts are public. Audit them.
The published grader and the per-question verdicts let you re-derive the score in one command. You need no engine and no network. The full methodology, the root-cause analysis and the 16 raw logs are open too.