NVIDIA paper: frequent context compaction keeps agent accuracy while cutting serving cost up to 7x
rohanpaul_ai · x · 2026-11-05
A new NVIDIA paper, "You don't need long context," shows per-token generation cost rises sharply with context length: serving open-weight models at 256K context costs 3.6x-7x more per generated token than 16K (measured on GLM 5.2, Kimi K2.6, Qwen 3 Coder Next at H100 on-demand pricing).
The fix: have agents stop at a set limit, write a short handoff note, and restart with only the task and that note; for long documents, search files with Bash tools instead of reading everything.
Across 6 long-horizon benchmarks, at most 5 of 22 compaction settings per model lost more than 2 points versus full history. Compacting after every action kept Claude Opus 4.5 at 70.2% on SWE-bench Verified vs 70.0% with full history. The authors question the need for long-context LLMs and outline "semantic state space models" (SSSMs).
More from Infra
- Beff backs space data centers: policy will only accelerate the Dyson swarm — beffjezos · 2026-11-05
- Texas datacenter moratorium sparks backlash over trillions in lost investment — beffjezos · 2026-11-05
- Researchers praise Together Compiler for open discussion of self-improving infra tradeoffs — simran_s_arora · 2026-11-05
- The compute arbitrage math: every $1 in data centers returns $4 — TinfoilTricorn · 2026-11-05
- Unsloth ships Qwen3.8-Flash-Next local guide: 125B MoE runs on 75GB RAM, 1-bit is 79% smaller — gnukeith · 2026-11-05
- Ripple ladder shows data beats architecture: 4x gains from new dataset vs 3x from new arch — finbarrtimbers · 2026-11-05