ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL
Zhuoshi Pan, Qizhi Pei, Junru Lu, Honglin Lin, H. Vicky Zhao, Di Yin, Xing Sun
EMNLP 2026 (Main Track)
cs.CL
2026-08-29
ContextPilot adds planning, long-term memory, and soft offloading tools, then spends extra RL rollouts on high-impact context edits. ContextPilot-8B-RL averages 69.40 on four long-context QA tasks, 3.55 points above StateLM-8B-RL.
Long-horizon agents grow a working context every turn. Rule-based truncation treats the model as a passenger. Proactive managers such as StateLM let the model search, delete, and summarize, but the toolset is thin: little planning, little long-term memory, little reversible compression. Training is coarser still. A context edit rewrites later history, and different edits have very different effects on final success, yet the same trajectory-level reward is copied onto every intermediate action.
ContextPilot targets three gaps: tools, exploration budget, and credit.
On top of StateLM-style search, deletion, and summarization, the toolset adds planning (plan, analyzeText, checkBudget), long-term memory (memorize writes entity-time-event items with edges; readMemory returns a node plus neighbors), and soft offloading (summarizeContext, LLMLingua-2 compressContext, and foldHistory, which collapses history into keywords plus a summary that search can recover). History-changing calls split a trajectory into snapshots used as the training unit.
SFT trajectories come from Qwen3.5-397B-A17B under a harness whose hints never enter the kept data. RL has two custom pieces. Context-aware partial rollout scores each management action by relative context-length change and entropy change versus the start of the trajectory, then spends leftover snapshot budget branching at the most sensitive edits. Fine-grained credit averages terminal rewards of every sampled continuation through a snapshot, then runs GRPO inside the snapshot group of one query.
Long-context QA starts from Qwen3-8B/14B and Gemma4-E4B-it. Deep search skips SFT and RL-tunes WebSailor-7B and WebExplorer-8B on 1,000 OpenSeeker items. Inference uses a 32K input cap.
Long-context QA, mean of three runs:
| Method | Window | NovelQA | ∞Bench | LongMemEval-S | BrowseComp+ | Avg |
| StateLM-8B-RL | 32K | 84.15 | 73.07 | 59.73 | 46.44 | 65.85 |
| ContextPilot-8B-RL | 32K | 83.88 | 75.25 | 64.27 | 54.18 | 69.40 |
| ContextPilot-14B-RL | 32K | 84.81 | 81.08 | 67.40 | 55.50 | 72.20 |
| Qwen3.5-397B + tools | 32K | 91.94 | 92.13 | 83.60 | 80.96 | 87.16 |
The 8B-RL model is 3.55 points above same-size StateLM and 7.74 points higher on BrowseComp+. RL barely moves NovelQA, whose other split was in SFT, but adds 5.34 points on BrowseComp+, whose inputs average about 552K tokens. Bare Qwen3-8B at 128K averages 45.93; the same tools without training drop to 27.61.
On deep search, ContextPilot on WebExplorer-8B averages 50.10 versus 49.09 for SUPO and 47.92 for OpenSeeker trained on the same 1K items without context tools. Working context on BrowseComp stabilizes near 8K-10K tokens per turn, against a baseline that grows toward 30K. Adding planning, soft offloading, then long-term memory on the teacher model lifts the average from 77.89 to 87.16, and BrowseComp+ from 63.49 to 80.96. Entropy-only branching is unstable; adding context variation plus snapshot credit takes Qwen3-8B from 65.78 SFT to 69.40.
A 32K managed window beating a 128K raw backbone says the scarce resource is often how the agent edits its own working memory, not another context-length SKU. Teams already training ReAct search agents can drop the toolset and snapshot credit into an existing pipeline.
The gains are incremental. The deep-search edge over SUPO is about 1.5 points. The clearer win is a shorter working context and tool use that looks like management rather than retrieval-only. Code and weights are public.
The authors say the toolset still misses some editing needs, hyperparameters were not searched hard, and evaluation never leaves long-context QA and deep search: no coding agents, no GUI agents. SFT used a constrained teacher; production will not. BrowseComp+ sits in the QA suite because it uses a fixed corpus with no live web, so it is not full internet search. Partial rollout costs extra samples, and that bill is not itemized.