New paper: compaction barely hurts agent benchmarks, but real user sessions tell a different story
maksym_andr · x · 2026-10-31
A new paper shows compaction doesn't hurt agent performance on frontier benchmarks like PaperBench, ProgramBench, or PostTrainBench — these tasks are surprisingly compaction-friendly, as agents reconstruct missing context from AGENTS.md, code repos, and web search. The gap with real-world complaints: users keep injecting constraints and preferences that agents struggle to track. Jeffrey Ma notes SWE evals are highly Markovian (the repo is the memory), while real agent use is multi-turn and partially observable — explaining why models feel "benchmaxxed."
More from coding & agent
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- Testing agents with explicit rejection records: a cheaper path to verified troubleshooting answers — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31
- PI agent rejected 75 proposals in 30 hours; manager banned recurring bad ideas — krishnan · 2026-10-31
- Next 9 proposals explored 7 new directions: rejection became information — krishnan · 2026-10-31