Harness Evolution Evaluation is Overestimated

UW · hf · 2026-07-18

This work re-examines the automatic harness evolution evaluation paradigm for LLM agents.

The authors point out that existing methods typically search for harness configurations using unit test feedback and report final scores on the same public benchmark, leading to two issues:

Conflating harness improvements with extra search: Harness evolution is itself an iterative search and should be fairly compared against simple task-level test-time scaling / discovery baselines under identical feedback and inference budgets.
Potential benchmark overfitting: Using the same tasks for both searching and final evaluation means improvements might just be memorization of that specific task set, rather than learning generalizable harness designs.

The authors conducted systematic experiments on Terminal-Bench 2.1 using GPT-5.4 and Claude Opus 4.6. Results show that:

Automatic harness evolution does not consistently outperform simple test-time scaling methods.
Generalization is limited on held-out tasks.
Current evaluation protocols likely overestimate the gains from automatic harness design.

Conclusion: Fairer evaluation protocols and more appropriate benchmarks are needed to assess the true value of automatic harness design. Code is open-sourced.

Original post →

Harness Evolution Evaluation is Overestimated

More from coding & agent