NVIDIA paper: frequent context compaction cuts LLM serving cost up to 7x vs long context
khademinori · x · 2026-11-21
NVIDIA researchers published "You don't need long context", questioning whether LLMs need huge context windows for long-running tasks.
- Cost measurements: growing context from 16K to 256K raises per-token serving cost by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next (computed at on-demand H100 prices)
- Alternative: frequent context compaction — keep a compact task-state representation and start a fresh context when needed; for long documents, keep them out of context and let the agent selectively read what it needs
- Findings: across a broad range of long-horizon tasks, models sacrifice little to no accuracy; performance on long-horizon software engineering holds even with at most one action per context, motivating the concept of semantic state space models (SSSMs)
The paper argues these savings rival or exceed other optimizations like aggressive quantization and structured sparsification, challenging the need for long-context LLMs.
More from Infra
- Analyst speculates OpenAI's problems stem from GPU shortage, no SpaceX access — mark_k · 2026-11-21
- Researcher builds local GPU server to run 24/7 AI agent system after Mac mini runs out of memory — nodosenlared · 2026-11-21
- Custom engine runs Inkling 2 Mini at 129.5 tok/s with 344K context on a single RTX 3090, 2.3x faster than llama.cpp — teortaxesTex · 2026-11-21
- Counterintuitive: KV cache quantization in llama.cpp uses MORE VRAM than f16 — Fancy-Snow7 · 2026-11-21
- Cohere launches confidential computing with encrypted inference; exec says rivals' ZDR labels are marketing — Nils_Reimers · 2026-11-21
- Gary Marcus shares chart of Oracle CDS spiking parabolically to record highs — GaryMarcus · 2026-11-21