Amazon paper: 54% of harness-evolution gains are benchmark overfitting
Wenxuan Ding · hf · 2026-10-21
Researchers from Amazon and NYU introduce Harness-Delta Attribution (HDA), an evaluation protocol that decomposes gains from agent harness evolution (optimizing prompts, skills, tools, and orchestration without weight updates) into artifact-driven overfitting, test-time-scaling gains, and residual generalizable improvement.
- Applied to 11 executor models × 3 harness-evolution frameworks across math, coding, creativity, and agentic tasks (21 settings).
- On average, overfitting explains 54% of search-set gains and test-time scaling 14%; combined they exceed 50% in 14 of 21 settings.
- Conclusion: most search-set gains from harness evolution do not reliably transfer to held-out tasks, implying systematic benchmark inflation.
More from Research
- beignet: A Neural Field Architecture with Multiresolution Fourier Pyramids for Fast PDE Solving — jon_barron · 2026-10-21
- Salesforce AI's SRD aligns agent foresight with hindsight at zero test-time cost — lupantech · 2026-10-21
- New Proof: Perfect VAE Reconstruction Can't Verify Encoder-Decoder Codebook Agreement — burny_tech · 2026-10-21
- Sander Dieleman on typicality: why likelihood fails for perceptual generative models — sedielem · 2026-10-21
- WildAWM: fully open-source agent task synthesis yields +10.58 OOD gains for Qwen3.6-35B — XuCanwen · 2026-10-21
- New paper: let models learn reward hacking, then knock out those parameters at deployment — EthanJPerez · 2026-10-21