NVIDIA team argues most tasks don't need long context: agents that compact stay under 50K tokens
HankYeomans · x · 2026-11-06
A post discussing an NVIDIA team's finding that most "long context" work doesn't actually need long context. Their agents keep compacting their own notes and stay well under 50,000 tokens, with accuracy barely dropping on coding, search, and long-document tasks.
The poster adds the cost angle: serving open models with a 256K-token window instead of 16K costs 3.6–7x more per output token, and on a DGX Spark, later tokens took about twice as long as early ones. Since long-context tokens are sold at a flat price, most users never notice the difference.
The wildest result: an agent that takes one action per context and carries only a short written summary forward matched full-context performance on SWE-bench Verified with Opus 4.5. The retweeter predicts the future lies in parallel-running compaction plus better integrated short-term memory systems, since hitting 1M tokens and then waiting minutes for compaction is a poor experience.
More from coding & agent
- Late to a talk, his AI agent tapped the live transcript and took notes for him — vincent_koc · 2026-11-06
- Anthropic engineer confirms Claude Code Desktop is in the works, asks for feedback — iamsahaj_xyz · 2026-11-06
- VoidZero 十月更新:Rust 版 TypeScript 提速 8 倍,SDK 开源 — cnakazawa · 2026-11-06
- One-shot in Claude: 10-min dinosaur documentary built with Higgsfield MCP — piotrbinkowski · 2026-11-06
- Voice-to-app demo: PlaiPin device + Grok Gadget SDK builds apps without a keyboard — tetsuoai · 2026-11-06
- Wavefunction Bench Tests Coding Agents on Quantum Chemistry: Only 3 of 9 Systems Passed — CatAstro_Piyush · 2026-11-06