Context Language Models paper: LLMs manage their own context, +11.4% accuracy with 21.5% fewer FLOPs
RulinShao · x · 2026-09-30
A new paper by Rulin Shao et al. (with Luke Zettlemoyer, Pang Wei Koh, Nathan Lambert and others) introduces Context Language Models (CLMs): LLMs that natively manage their own context by treating it as a file the model can freely update — learning what to keep, and extending naturally to multi-agent setups where each agent's context is a file.
Key results:
- Zero-shot CLMs built on existing models beat SOTA context-management strategies: +11.4% accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, +5% with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement on a 24-hour multi-repo agent-swarm task at equal compute.
- Moving context management from external harness to intrinsic behavior enables steering via natural-language instructions evolved in a skill-optimization loop (+35.9 points held-out on a context-management task).
- An online RL method lifts Qwen3.5-9B by 47.6% on BrowseComp-Plus with 12% fewer FLOPs.
- Co-designed Suffix Cache Reuse cuts serving compute 35% vs standard SGLang.
Paper and code are public.
More from coding & agent
- Dev argues Rust would be better if modules, not crates, were the compilation unit — zetalyrae · 2026-10-25
- Dev open-sources BizAgent framework where one AI lead agent manages a dozen AI employees — Low-Start2333 · 2026-10-25
- Formal Verification Yields to Static Analysis, Which Yields to Better Language Design — xennygrimmato_ · 2026-10-25
- Scanning 115,798 agent ledger rows: zero capability revocations, none with expiry — Affectionate-Sun265 · 2026-10-25
- 'Slurm's defense remains annoyingly strong': dev watches agents hit HPC wall — sloppenheimer · 2026-10-25
- 5 open-source tools turn user interviews into a full content marketing workflow — sven_ai · 2026-10-25