Reproducing Google's Harness2 Lifts Qwen3-9B Pass Rate from 31% to 45%
yb2698 · x · 2026-10-24
Applying Google's harness2 technique—Claude Code subagents evolve a custom harness (tools + system prompt) per task—the author lifted Qwen3-9B's mean pass rate on a TB-Lite subset from 31% to 45% across 8 tasks.
- Each task starts from the same bare IPython harness (0 tools, 278-word prompt); subagents, given no solutions, wrote 13 Python tools across 7 tasks
- Big wins: protein-sequence 3→10, corrupted-filesystem 5→10 (of 10 runs); but floor-plan fell 5→0 and sakila-sql 7→4—per-task degradation is real
- Key finding: more tools don't help. Model-defined tools: −2.5 pts; one shared generic harness: −2.5 pts; per-task evolved harness: +13.8 pts. Correlation between tool count and gain: −0.15—the right tool matters, not the number
Related event: Reproducing Google's harness2: Qwen3-9B pass rate jumps from 31% to 45%(6 posts)→
More from coding & agent
- GPT SOL 6.1 builds a 3D axe from scratch via Blender MCP in 7 minutes — OpenAIDevs · 2026-10-24
- icons.computer indexes 3,000+ curated app icons, agent-ready via Firecrawl — devdigest · 2026-10-24
- Dev: With LLMs Helping Setup, There's Never Been an Easier Time to Daily Drive Linux — KevinNaughtonJr · 2026-10-24
- Postman's first AI-Ready API leaderboard puts You.com (96) above OpenRouter (94) — RichardSocher · 2026-10-24
- Richard Socher launches Flint, an open-weights local coding model hitting 125+ tokens/s on a MacBook — RichardSocher · 2026-10-24
- Agent platform Outpost teases mobile app to manage all your agents from one place — jacob_posel · 2026-10-24