NVIDIA paper: frequent context compaction costs little accuracy, questioning long-context LLMs

rohanpaul_ai · x · 2026-11-05

A NVIDIA research team's paper "You don't need long context" (arXiv:2611.05306) argues long-context LLMs may be unnecessary.

Key findings:

Later tokens cost more to generate: serving open models at 256K context costs 3.6x–7x more per token than 16K.
With frequent compaction — agents write a short handoff note at a set limit and restart with only the task and that note — frontier models sacrifice little to no accuracy across long-horizon, long-input tasks.
LLMs don't collapse on long-horizon software engineering tasks even when taking at most one action per context.
The authors outline the concept of semantic state space models (SSSMs).

More from Infra