Memory layer cuts agent context tokens 23-62x and beats full history on 90-day recall
No_Advertising2536 · reddit · 2026-09-16
A memory API builder published a rare measured A/B on Reddit: across three synthetic dialogue corpora (companion, support, coding) with planted facts and ground truth, comparing a memory layer (per-day fact extraction into a 600-token budget) vs sending full history, with gpt-4o-mini as the answer model.
Key numbers
- Context tokens: full history grows linearly (6-7.5k per request at 90 days); memory stays flat at 120-330 tokens — 9x to 62x compression
- Cost: at $0.15/M input tokens, 1,000 users × 20 req/day means $630/month for full history at 90 days vs $19 with memory
- Old-fact recall: at 90 days memory actually wins — companion 1.00 vs 0.88, support 0.75 vs 0.38. "Just send everything" degrades with length too
Three surprises
- Memory flopped badly on support at 30 days (recall 0.25): the extractor stored "has a loyalty number" instead of the actual ID — identifiers and named entities are memory's weak spot
- Full history isn't a free lunch: the model finds month-old facts in 7.5k tokens only 38% of the time
- Measuring caught two bugs in the author's own salience filter, which was dropping facts phrased as reported speech ("pinned to Python 3.12"); fixing it lifted coding recall 0.75→0.88
The bench, corpus generator (fixed seed), and cost report are all open-sourced.
More from coding & agent
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- She fired herself from her own Instagram: AI agent runs the account, views up 82.3% — heyshrutimishra · 2026-11-02
- Sourcegraph: AI writes code faster than teams can review it, driving more outages — AI Engineer · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02
- GitHub launches Agent HQ: Anthropic, OpenAI, Google coding agents in Copilot — emmanuelvivier · 2026-11-02
- DeepSeek open-sources Super-Agent, a coding agent that reads team chats to minimize human input — emmanuelvivier · 2026-11-02