A multiplier cannot rescue a zero
Ask a memory system “What shift was Admon assigned on Sunday?” and only one thing matters: can it find the single session that contains Admon?
Ours couldn’t. Not occasionally — structurally.
The blind spot
Mnemosyne OS ranks memory by dense-vector similarity: e5 embeddings, cosine, one ranking. That is the right tool for meaning — “the time I complained about the hotel” finds the complaint without sharing a word with it. But a rare literal token — a proper noun, an identifier, a version number — is precisely what an embedder maps close to noise. It has never seen that string. It has no meaning to hang on it. The one session you need can sit at rank #800.
Measured on LongMemEval full-haystack — same protocol as the 72.9% campaign, roughly 480 distractor sessions per question — on a 48-question sample: for 6 questions out of 48, not a single evidence session came back. Among the 35 questions where we could track the exact answer-bearing chunk, it was absent from the served context 10 times.
You cannot answer with what retrieval never handed you.
The seductive fix, measured to nothing
The codebase already contained the obvious answer: a term boost. Extract the distinctive tokens from the question, and when a memory chunk contains one verbatim, multiply its cosine by 2.
Two things happened, both instructive.
First, fed naively, it was a catastrophe. A prose question capitalises its first word, so “What shift was Admon…” boosted “what” — a token present in nearly every conversational chunk in existence — and evidence recall collapsed from 38/48 to 17/48. The rule that survived: a capital letter only evidences a proper noun when the word is not sentence-initial.
Second, once tamed, it measured neutral end-to-end. The arithmetic says why. The sessions we were missing scored near-zero cosine, and 0.30 doubled is still below 0.80. A multiplier amplifies a signal that exists; it cannot create one. The chunks this blind spot loses are not ranked low — they are ranked nowhere.
A second ranking, not a bigger multiplier
The fix that worked never consults the vectors at all.
Rank the vault twice. Once by cosine, deeper than the budget (200 instead of
32). Once lexically, with Okapi BM25 — the boring, forty-year-old term-weighting
scheme that managed “hybrid search” products run under the hood. Then merge the
two rankings with Reciprocal Rank Fusion: each document scores the sum of
1/(k + rank) across channels.
Ranks — never scores. A cosine lives in [0,1]; a BM25 sum is unbounded. The classic way hybrid search goes wrong is trying to normalise one against the other. RRF never compares them.
Fusion by rank also carries a safety property worth knowing before reading any result: a document that appears in only one ranking can at best tie the other channel’s #1 — it can never outrank a document both channels agree on. The lexical channel cannot seize the budget; it can only displace vector hits ranked so deep that their own reciprocal has decayed. And since BM25 returns nothing for a document sharing no query term, the channels overlap only partially — which is exactly where fusion has something to add.
The context budget did not move: still 32 sources. This changes which chunks are served, never how many.
The numbers
| Measurement (48-question full-haystack sample) | Vector only | + lexical channel |
|---|---|---|
| Questions with all evidence sessions retrieved | 38/48 | 41/48 (+6/−3) |
| Answer-bearing chunk in the served context (35 tracked) | 25/35 | 30/35 (+6/−1) |
| End-to-end, strict judge | 29/48 | 37/48 (paired p = 0.0215) |
Retrieval is measured deterministically — byte-identical inputs, no LLM in the loop, free to reproduce. The end-to-end number goes through an LLM judge, which is noisy: replaying byte-identical runs through this rig flips about 2.6 verdicts per 48, so a gap under ~5 questions is not resolvable by this bench. That is why retrieval is the instrument and the answer rate is the confirmation. The +8 clears the floor with room to spare — and it reproduced across two independent runs that agreed verdict for verdict.
Audit it. Every number above recomputes from published per-question rows:
the raw runs (both strict runs, the deterministic recall files, the holdout)
and the mechanically extracted ledgers live on the
benchmark provenance page,
with the campaign’s reading key in
lexical-2026-08/.
One command — node verify.js — re-derives every cell and fails on any mismatch.
The part where we try to fool ourselves, and fail
There are two classic ways to manufacture a result like this: tune the hyper-parameters on your sample, and report only the sample you developed on.
We did neither.
The parameters are the literature defaults — BM25 k1=1.5, b=0.75; RRF
k=60, the constant from the original paper. Deliberately untuned, because our
48-question sample runs about 13 points easier than the benchmark it is drawn
from, and fitting to an easy sample manufactures gains that do not transfer.
Then the holdout: 48 fresh questions the channel had never seen, drawn after the design was frozen. The gain transferred — +4 questions with complete evidence, +2 answer-bearing chunks, zero regressions. Not one question retrieved worse. Same dominant gain category (temporal reasoning) as the development sample.
What we refuse to say
The context for this work: earlier in August we measured our own retrieval montage augmented with a managed hybrid-search API (Google Vertex AI Search). That Google-augmented montage scored 37/48 on the strict judge. The local channel now reaches the same 37/48 — with no API, no account, and nothing leaving your machine.
Read that sentence carefully, because of what it does not say. It does not say we beat anyone. We compared two of our own montages — one calling an external service, one not — under one protocol we control. The managed product was never scored in a way that would permit a product-versus-product claim, and we will not pretend otherwise.
The other reservations stay printed next to the number, where they belong: one benchmark; a 48-question sample that is thinner and easier than its parent set; an answer-side judge with a known noise floor. The holdout removed the objection we feared most — that we had quietly fit our own development set — but a second benchmark family is still owed.
The boring parts that make it real
- The inverted index is persistent — plain SQLite tables inside the vault database, a term dictionary keyed on integers. Cold build on a 2,470-chunk vault: 19.5 s naive (one fsync per posting), under a second in a single transaction. Incremental afterwards: an ingest indexes only what it added.
- Not FTS5, deliberately. The benchmark harness runs on
node:sqlite, which has no FTS5 module — the lever would have been unmeasurable there, and an unmeasured lever is indistinguishable from one that never ran. FTS5’s defaults (k1=1.2, theunicode61tokenizer) would also have silently changed the very ranking we measured. - An incomplete index returns nothing, and the pipeline falls back to the vector ranking — yesterday’s behaviour. A half-built index has wrong document frequencies and produces a confident, silently wrong ranking. A missing feature beats a wrong answer.
- Per-app memory isolation is enforced inside the channel itself, with its own tests — not delegated to whoever happens to call it.
The channel ships in v1.3.8, on by default. MNEMO_LEXICAL_FUSION=0 restores
the vector-only path byte for byte — the escape hatch our own reservations
demand.
What we can say fits on one line: a proper noun your memory could not hear now finds its session — locally, measured, reproduced twice, and confirmed on questions it had never seen.