Harness Evolution Evaluation is Overestimated
UW · hf · 2026-07-18
This work re-examines the automatic harness evolution evaluation paradigm for LLM agents.
The authors point out that existing methods typically search for harness configurations using unit test feedback and report final scores on the same public benchmark, leading to two issues:
- Conflating harness improvements with extra search: Harness evolution is itself an iterative search and should be fairly compared against simple task-level test-time scaling / discovery baselines under identical feedback and inference budgets.
- Potential benchmark overfitting: Using the same tasks for both searching and final evaluation means improvements might just be memorization of that specific task set, rather than learning generalizable harness designs.
The authors conducted systematic experiments on Terminal-Bench 2.1 using GPT-5.4 and Claude Opus 4.6. Results show that:
- Automatic harness evolution does not consistently outperform simple test-time scaling methods.
- Generalization is limited on held-out tasks.
- Current evaluation protocols likely overestimate the gains from automatic harness design.
Conclusion: Fairer evaluation protocols and more appropriate benchmarks are needed to assess the true value of automatic harness design. Code is open-sourced.
Related event: Agent Harness Self-Improvement and Domain-Specific Design(6 posts)→
More from coding & agent
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- Stop building agents that can't change models: stable contracts, external state, real evals — rseroter · 2026-10-27
- Cheap models can cost more in agent harnesses than frontier ones, researcher warns — vasuman · 2026-10-27
- Kanzo: A 5k-Line Rust Kanban Binary With MCP for Claude Code and Codex — allmudi · 2026-10-27
- 'A human approves everything' is a trap: approval fatigue and how to actually design safe agents — Efistoffeles · 2026-10-27