Six months of evals: never let the model rewrite the source; BM25 weighting lifts recall@10 to 0.86

Cryvixx · reddit · 2026-08-24

The author built Silica, a harness that gates LLM writes to a markdown vault — the model proposes edits, a parser and state machine verify them, notes are read back, and failures are reverted; the read path needs no model at all. Findings from 50+ papers applied to real evals (small samples, judge is a local model — treat numbers as directions):

The one replicated result: do not let the model rewrite the source. Verbatim notes beat summarized ones across two conversations (0.592 vs 0.428; 0.840 vs 0.693), with the summarizing arm last in every question category.
Weighting the co-occurrence graph with BM25 instead of raw counts: recall@10 went 0.51 → 0.86. Once that leg is scored correctly, fusion weights and the RRF constant stop mattering — the knobs were only compensating for a badly scored leg.
An obviously useful retrieval leg recovered zero pairs: note-to-note derived edges added nothing beyond embeddings; dropping it moved recall@10 from 0.8233 to 0.8815 and cut cost from 5 ms to 1.7 ms.
A 0.669 FActScore turned out to be a harness bug, not hallucination: after fixing the judge reference, scores hit 0.999/0.991/0.961 across 5,855 facts.
A benchmark table with no bad rows is a marketing asset: weak results (0.32 folder agreement) ship unedited, and the runner refuses cross-config comparisons.

Applications: 200 PDFs become 200 resolvable, individually revertible transactions; note relations route over wikilinks plus a concept graph; the vault clusters into communities; the calendar is just the vault read along dates.

Original post →

Six months of evals: never let the model rewrite the source; BM25 weighting lifts recall@10 to 0.86

More from coding & agent