Tailored harness lifts a small model on 4 of 8 tasks; tool count correlates −0.15 with gains
yb2698 · x · 2026-10-24
The author shares agent eval results over 8 tasks × 10 runs: 4 tasks improved by 4–7 runs, 3 regressed, 1 was flat. The takeaway is that a tailored harness can unlock a lot from a small model, but with only 8 tasks × 10 runs and wide spread, this is a signal to scale up rather than a final answer.
Crucially, you can't just count tools: floor-plan got 3 tools and a 452-word prompt for −5 runs, while protein-sequence got 1 tool for +7 runs. The correlation between tools added and gain is −0.15 — it's the right tool that counts, not the number.
Related event: Reproducing Google's harness2: Qwen3-9B pass rate jumps from 31% to 45%(6 posts)→
More from coding & agent
- PM's step-by-step guide to building product strategy with Claude Code in 2 hours — aakashgupta · 2026-10-24
- Six commonly confused LLM API terms: context window, temperature, reasoning tokens — ClickOk5811 · 2026-10-24
- Mubit trains proprietary models from your agent traces so you stop being an API tenant — SucceededMind · 2026-10-24
- Microsoft Open-Sources Frontier Tour Resource Hub with Code, Demos, and Labs — WirelessLife · 2026-10-24
- Rintawa: a solo dev's open-source Rust engine for persistent, simulated AI worlds — march-taylor · 2026-10-24
- Microsoft & HF turn database checks into RL rewards, lifting Qwen3.8-27B agents from 51.7% to 60.78% — SergioPaniego · 2026-10-24