New paper: benchmarks are surprisingly compaction-friendly, no method beats self-summarization

khademinori · x · 2026-11-04

A new arXiv paper systematically evaluates context compaction on real long-horizon agentic tasks for the first time: the authors fix the OpenCode scaffold, implement five compaction strategies, and test three frontier models on PaperBench, ProgramBench, and PostTrainBench.

Key findings:

For 1M-context models, the compaction threshold has no measurable effect; GPT-5.3-Codex (smallest context) degrades at lower thresholds on PaperBench
Counterintuitively, compaction method and threshold mostly don't affect performance; nothing beats plain self-summarization
But compaction silently drops many initial rules — and preserving constraints discovered later, or legitimate rule changes, remains unsolved
The authors call for new agentic benchmarks where agents learn/override/conflict rules through environment interaction, since compaction may discard the agent's own uncertainty about rule violations

Original post →

New paper: benchmarks are surprisingly compaction-friendly, no method beats self-summarization

More from coding & agent