Memory layer cuts agent context tokens 23-62x and beats full history on 90-day recall
No_Advertising2536 · reddit · 2026-09-16
A memory API builder published a rare measured A/B on Reddit: across three synthetic dialogue corpora (companion, support, coding) with planted facts and ground truth, comparing a memory layer (per-day fact extraction into a 600-token budget) vs sending full history, with gpt-4o-mini as the answer model.
Key numbers
- Context tokens: full history grows linearly (6-7.5k per request at 90 days); memory stays flat at 120-330 tokens — 9x to 62x compression
- Cost: at $0.15/M input tokens, 1,000 users × 20 req/day means $630/month for full history at 90 days vs $19 with memory
- Old-fact recall: at 90 days memory actually wins — companion 1.00 vs 0.88, support 0.75 vs 0.38. "Just send everything" degrades with length too
Three surprises
- Memory flopped badly on support at 30 days (recall 0.25): the extractor stored "has a loyalty number" instead of the actual ID — identifiers and named entities are memory's weak spot
- Full history isn't a free lunch: the model finds month-old facts in 7.5k tokens only 38% of the time
- Measuring caught two bugs in the author's own salience filter, which was dropping facts phrased as reported speech ("pinned to Python 3.12"); fixing it lifted coding recall 0.75→0.88
The bench, corpus generator (fixed seed), and cost report are all open-sourced.
More from coding & agent
- Tinysarf: 245KB In-Browser Arabic Morphological Analyzers on WebGPU — shuding · 2026-09-16
- A 35k-Parameter WebGPU Model Turns English Into Cron Expressions in 31KB — shuding · 2026-09-16
- gpu-query: Parsing Data in the Browser With Tiny GPU-Run Models — shuding · 2026-09-16
- User's Grok bot takes over their LinkedIn and starts replying to lowballing recruiters — IndraVahan · 2026-09-16
- Codex now stops all threads once you hit your usage limit — NielsRogge · 2026-09-16
- Awesome Robot Use Agent: A Curated Collection of Papers, Tools and Demos for Robot Agents — AdinaYakup · 2026-09-16