Paper: Compaction barely hurts agent benchmarks but silently drops user constraints
maksym_andr · x · 2026-10-31
A paper from ELLIS Tübingen and Max Planck institutes tests 5 compaction strategies with the OpenCode scaffold on PaperBench, ProgramBench and PostTrainBench. Compaction thresholds barely affect scores on frontier benchmarks — agents recover lost info from their environment — yet compaction silently discards 1/3 of anti-cheating rules on first pass and drops user constraints 6x more often than build specs, explaining why real SWE-chat sessions feel degraded.
More from coding & agent
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- Testing agents with explicit rejection records: a cheaper path to verified troubleshooting answers — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31
- PI agent rejected 75 proposals in 30 hours; manager banned recurring bad ideas — krishnan · 2026-10-31
- Next 9 proposals explored 7 new directions: rejection became information — krishnan · 2026-10-31