Context Language Models
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh
cs.AI, cs.CL, cs.LG
2026-09-29
UW and Meta make live context an editable file. Zero-shot CLMs beat the best compaction baseline by 11.4% relative accuracy on BrowseComp-Plus at 21.5% fewer FLOPs.
Long-horizon agents keep appending. Most systems still park context policy in the harness: compact at a length threshold, or expose a fixed menu of compaction, offload, and retrieval. The model may choose when to fire a tool. It cannot invent a new operation. Recursive Language Models put a long input in a REPL variable and read it on demand, but whatever they pull still concatenates onto the live window, so the window only grows.
Researchers at the University of Washington, Meta Superintelligence Labs, MIT, and Trillium Labs hand the whole transition to the model. A standard LM concatenates new tokens each turn. A Context Language Model (CLM) writes the next context outright.
The live context is mirrored to a file whose path sits in the system prompt. The model edits that file with ordinary Bash; the harness syncs it as the next prompt. No edit means append. Extra context files are extra agents: create one to spawn a subagent, delete it to tear the subagent down.
Traces grow their own tactics. One run kept a subagent scoreboard with 163 in-place edits and held the window at 6-8K tokens, invented a notes role, and defined compactturns, calling it 37 times.
A single instruction changes policy. On BrowseComp-Plus with Claude 4.6 Sonnet, compacting to about 4K after passing Y tokens moved the median first-compaction size with Y in {16K, 24K, 32K}. Compacting only at subtask boundaries raised within-two-turn compaction from 0.40 to 0.88. Backing up the file before every edit raised full backups from 0 to 0.77. A prompt-evolution loop writes a skill document from training rollouts and keeps winners on a development split: assisted evolution uses Qwen3.6-27B with Claude Fable 5.1 as proposer, self-evolution uses Opus 5 for both. Held-out ContextBench accuracy rises by up to 35.9 points at lower compute. Parametric learning uses stepwise GRPO, copying the trajectory outcome advantage onto every segment, plus a success-gated efficiency term that re-ranks only successful group members by prefix-reuse FLOPs. Failed trajectories get a zero efficiency advantage, so the model cannot farm the bonus by deleting useful context.
Mid-context edits break prefix cache. The paper counts prefix-reuse FLOPs: prefill from the first mismatch through the suffix, plus decode. Suffix Cache Reuse keeps KV for surviving suffix tokens, re-rotates RoPE, and prefills only the inserted span, relocating at most K=6 longest spans. Those states were computed under the old prefix, so the reuse is approximate.
The main table is zero-shot Qwen3.6-27B at 32K against Codex-style summary (trigger at 75% of budget), MEM1, Self-Compact, ACM, and RLM, all on Mini-SWE-Agent. BrowseComp-Plus has 830 questions.
| Task | CLM | Baseline |
| BrowseComp-Plus accuracy | 59.4% | 11.4% relative over Codex summary |
| Same, FLOPs | 21.5% below summary, 28.9% below MEM1 | |
| TerminalBench 2.1 | tied with summary | 70% of summary FLOPs |
| TBLite | 73.7% | summary 67.0%, CLM at 91% of its FLOPs |
On four math-optimization problems with Claude 4.6 Sonnet, 32K, and a cap of 100 scored attempts or five hours, CLM posts the best run on every task versus OpenEvolve: circle packing 2.618 vs 2.541 (+3.0%), Heilbronn 0.03653 vs 0.03127 (+16.8%). The subagent variant reaches 2.636 on packing. One run per problem.
On 12-hour EdgeBench-10, Qwen3.6-27B CLM scores 44.6 at 179 prefix-reuse PFLOPs against summary at 42.3 and 437 PFLOPs, about 5% higher score and 59% less compute. Claude 4.6 Sonnet reaches 51.0 versus 42.3 for summary. Subagents add almost nothing on this single-repo suite (44.2 / 50.4).
Software World runs six agents for 24 hours on requests, urllib3, and four downstream packages, with GPT-5.6-Sol at 272K. The score is geometric-mean instruction speedup on 17 CPU benchmarks from four packages the agents never see. At matched API spend, the CLM swarm's end-to-end speedup over the initial releases is 65% greater than the summary swarm.
ContextBench isolates management with needle retention, in-place Sudoku, KV Store, and Log Triage, at pressure up to 24× a 32K window. Fixed policies drop or fabricate needles, rewrite the whole board when they cannot edit in place, or offload values they then cannot evict from the live window. No fixed baseline is perfect on these toy tasks.
RL on OpenResearcher lifts Qwen3.5-9B CLM on BrowseComp-Plus from 28.8% to 42.5% (+47.6% relative) and cuts FLOPs from 1.52 to 1.34 (12% less). The same recipe takes summary from 34.7% to 42.1%. After training, CLM is 0.4 points higher at 38.8% fewer FLOPs. Before training the 9B CLM sat six points behind summary; unconstrained edits hurt when the model cannot manage them.
SCR matches standard SGLang at 60.2% on BrowseComp-Plus and drops empirical FLOPs from 10.98 to 7.14, a 35% server-side cut. Of the extra 7.8% of prompt tokens SCR reuses, 5.3 points come from reasoning tokens stripped by the chat template and 2.5 from model-driven edits.
The usable claim is small and sharp. Wrapping an existing model with a context-as-a-file layer already beats purpose-built compaction tools on deep research, terminal coding, 12-hour repo tuning, and a 24-hour multi-repo swarm, while spending less on re-prefill. Policy can start as a sentence, then a skill document, then GRPO with an efficiency gate. The code is public. SCR also pays off for ordinary reasoning endpoints, because stripping chain-of-thought already forces suffix re-prefill.
This is not a new backbone. It moves procedural memory out of the harness and lets the model search for cheaper management than the human prior. Subagents did not help on single-repo EdgeBench, and a small model without RL lost to summary. Do not swap it in as a universal drop-in.
Writable context is a new injection path. The paper cites OpenAI's finding that models already plant unauthorized instructions inside compaction summaries and then follow them. CLM lets the model rewrite the entire live window, so that path is wider.
The protocol is kinder to CLM than the prose admits. On BrowseComp-Plus, editing turns do not count against the 100-turn cap, and over-budget calls roll back and retry up to six times. EdgeBench allows 50 rollbacks. Whether every baseline got the same free edits and retries is not aligned in the main text. Math problems are one run each. The 65% Software World gap has no variance.
SCR reuses KV computed under the pre-edit prefix. On Qwen3.6-27B, 48 of 64 layers are linear attention and do not even recompute the inserted span. A 64-question sweep over K did not show a drop; the paper does not isolate SCR quality on the full 830. SGLang stores hybrid recurrent state only at request boundaries, so prefix hits are already incomplete. The authors say a chunk of leftover prefill is that serving limitation, not something SCR can fix.
Every ContextBench baseline got a method-specific skill written at CLM-like detail. That isolates tool use. It also means the diagnostic wins do not read as wins with no teaching. CLM assumes the model can already write code and plan.