Per-Task Evolved Harnesses Gain 13.8 Points While Generic Ones Lose 2.5
yb2698 · x · 2026-10-24
Ablation from the harness2 reproduction thread: letting the model define its own tools mid-run costs −2.5 points, one shared generic harness costs −2.5 points, but a harness evolved per task (harness2) gains +13.8 points. Tailoring the harness to each task is what matters, even for small models.
Related event: Reproducing Google's harness2: Qwen3-9B pass rate jumps from 31% to 45%(6 posts)→
More from coding & agent
- A 2-person AI team built its own video tool and dropped Zoom entirely — danshipper · 2026-10-24
- Agent posting safety: publish, fetch as a stranger, then assert on what's actually there — petrusenko_max · 2026-10-24
- Testing your agent's public output: publish, fetch logged-out, then assert — petrusenko_max · 2026-10-24
- 100% test coverage, zero bugs found: why this dev added live-post verification — petrusenko_max · 2026-10-24
- Claude approved its own broken 3D screenshots — how a 480Hz verification gate fixed agent self-grading — youthanasia669 · 2026-10-24
- PM's step-by-step guide to building product strategy with Claude Code in 2 hours — aakashgupta · 2026-10-24