847 agent runs show instruction-following falls from 94% to 41% as context fills
Unique-Werewolf-2784 · reddit · 2026-09-23
A developer tracked 847 agent runs and found instruction-following starts at 94% but collapses to 41% as the context window fills — a cliff, not a slope. Attention is U-shaped, so bigger windows just create a bigger ignored middle; some teams waste 60%+ of token budget re-injecting stale context. Aggressive compaction backfires: compressing an 18,282-token conversation to 122 tokens performed worse than no memory at all, since summaries keep the story but drop actionable facts. The author also cites a silent failure: 3,000+ docs in a memory layer, dashboard showed 91 memories, yet every search returned empty with no error. Their fix (Synap): retrieval-based memory across vector/graph/file stores under a token budget, with compaction checks reporting fact-survival rates, P75 latency under 15ms, and roughly halved token spend. Suggested self-check: measure accuracy at turn 5 vs turn 50 on the same task, then diagnose whether it's the attention cliff or lossy compaction. (Disclosure: author works on Maximem Synap; the 847-run data is from another r/LocalLLaMA developer.)
More from coding & agent
- Microsoft's Agensh Scales Multi-Agent Systems to 1,024 Agents Without a Central Orchestrator, Boosting Test-Pass Rate to 55% — andrew_n_carr · 2026-09-23
- Viral demo claims 'GPT-6' can drive browser Paint to draw, unverified — alexcovo_eth · 2026-09-23
- Parallel coding agents merge cleanly and silently break every test — RunAI_Coder · 2026-09-23
- Getting phone-captured text into your computer: OCR, vision models, agentic pipelines — silenceimpaired · 2026-09-23
- Free Bots: a persistent 3D city where AI agents work, earn, buy land and build houses — Daniel_Farinax · 2026-09-23
- Game dev looks like the programming field most resistant to AI — how much is it actually used? — marktenenholtz · 2026-09-23