FrontierHarness eval: 12 agent harnesses, same model — pass rates span 50%–67%, cost per pass $1.05–$18.34
stuffyokodraws · x · 2026-09-03
- The FrontierHarness eval argues the key question has shifted from "which model" to "which harness."
- Setup: 12 harnesses (Pi, Exo, Claude Code, Codex, DeepSeek Harness and more) tested on the same model, tasks and runtime — 360 runs, 2 billion tokens.
- Results: pass rates ranged from 50% to 67%, while cost per pass varied from $1.05 to $18.34 — harness engineering now matters as much as model choice for both quality and cost.
Related event: FrontierHarness: Same Model, 17-Point Gap Across Agent Harnesses(2 posts)→
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Pi Durable went through 7 rewrites yet agent session files still lack a standard — ricklamers · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18