Reproducing Google's harness2: Qwen3-9B pass rate jumps from 31% to 45%

@yb2698 reproduced Google's harness2 approach: letting Claude Code sub-agents evolve a custom harness per task. Testing Qwen3-9B on a TB-Lite subset (8 tasks × 10 runs), average pass rate rose from 31% to 45%. Ablations and per-task breakdowns show that task-tailored harnesses are the key driver, while tool count itself has almost no bearing on gains. The author stresses the small sample and high variance, framing this as a signal that needs larger-scale validation.

Confirmed

• All tasks started from the same bare harness: an IPython kernel, 0 tools, a 278-word prompt; Claude Code sub-agents wrote custom Python tools and new system prompts for 7 tasks, producing 13 custom tools in total
• Overall effect: average pass rate across 8 tasks rose from 31% to 45%
• Ablations: custom tools at runtime −2.5 points; a single generic harness shared across all tasks −2.5 points; per-task evolved custom harnesses (harness2) +13.8 points
• Per-task results (passes over 10 runs): protein-sequence 3→10, corrupted-filesystem 5→10, chained-forensic 1→5; 4 tasks improved sharply, 3 regressed, floor-plan dropped to zero, and 1 task was flat (4 tasks each gained 4–7 runs)
• Correlation between tool count and pass-rate gain is just −0.15: floor-plan gained 3 tools and a 452-word prompt yet passed 5 fewer runs, while protein-sequence added just 1 tool and improved dramatically

Why It Matters

• Ablations show the gains come from per-task harness tailoring, not tool stacking, providing reproducible evidence for agent benchmark design
• The results show custom harnesses can squeeze substantial capability out of small models, and harness choice significantly affects measured model performance
• The author's own caveat: small sample, high variance—treat this as a signal requiring larger-scale validation, not a conclusion

2026-10-24 ~ 2026-10-24 · 6 related posts

• [source] Reproducing Google's Harness2 Lifts Qwen3-9B Pass Rate from 31% to 45% — yb2698 · 2026-10-24
• How Claude Code Subagents Wrote 13 Custom Tools Across 7 Tasks — yb2698 · 2026-10-24
• Harness2 Reproduction: 4 Tasks Soar While floor-plan Drops to Zero — yb2698 · 2026-10-24
• [source] Per-Task Evolved Harnesses Gain 13.8 Points While Generic Ones Lose 2.5 — yb2698 · 2026-10-24
• [source] Tool Count Doesn't Predict Agent Gains: Correlation Is Just −0.15 — yb2698 · 2026-10-24
• Tailored harness lifts a small model on 4 of 8 tasks; tool count correlates −0.15 with gains — yb2698 · 2026-10-24