New paper: benchmarks are surprisingly compaction-friendly, but user constraints get lost

ELLIS Tübingen and other institutions have released "Benchmarks Are Surprisingly Compaction-Friendly," the first systematic evaluation of how context compaction affects long-horizon agents. Using a fixed OpenCode scaffold, the team tested 5 compaction strategies and concluded that on frontier benchmarks such as PaperBench, ProgramBench, and PostTrainBench, compaction barely affects scores — these tasks are unexpectedly "compaction-friendly."

Confirmed

• The paper examines whether compaction is a major obstacle on the road to AGI, and found no substantial performance impact on frontier benchmarks (consistently reported across multiple posts by maksymandr and PandaAshwinee)
• Experiments covered models including Claude, GPT, and Gemini, examining which information gets pruned first during long tasks
• Constraint-style instructions are 6 times more likely to be forgotten than goal-type content in long tasks: models tend to cut information deemed "unnecessary for task progress," and users' verbal instructions are often the first to go
• The work has also drawn attention in Japan (per maksymandr)

Why it matters

• The results suggest frontier benchmarks may mask the true cost of compaction: benchmark tasks lack user constraints that need to be remembered long-term, making them insensitive to compression
• For real-world agent products, "soft constraints" like user preferences and boundary conditions are silently dropped over long sessions — a weakness that benchmarks struggle to capture; the authors, including @maksymandr, argue compaction remains a weak point for agents in real conversations

2026-10-31 ~ 2026-10-31 · 5 related posts

• [source] Paper: Compaction barely hurts agent benchmarks but silently drops user constraints — maksym_andr · 2026-10-31
• [source] New paper tests 5 compaction strategies: long-horizon agent benchmarks are surprisingly compaction-friendly — maksym_andr · 2026-10-31
• [source] Compaction study: agent constraints vanish 6x more often than task goals in long sessions — maksym_andr · 2026-10-31
• New paper: compaction barely hurts agent benchmarks, but real user sessions tell a different story — maksym_andr · 2026-10-31
• Paper finds context compaction doesn't hurt agent performance on frontier benchmarks — PandaAshwinee · 2026-10-31