Same model, 17x cost gap: FrontierHarness benchmarks 9 agent harnesses across 360 runs
Double-Entertainer62 · reddit · 2026-09-10
FrontierHarness tested 9 agent harnesses (Pi, Exo, Claude Code, Codex, DeepSeek Harness and more) on the same model, tasks and runtime: 360 runs, 2B tokens. Pass rates ranged 50–67%; Claude Code and DSH Creator tied at 19/30. Median cost per pass varied from $1.05 to $18.34 — a 17x gap. Key insight: cheap successful runs can hide expensive workflows; failed attempts, retries and manual cleanup count toward total spend, so track total spend divided by accepted tasks, plus time and human intervention. Recommendations: Codex for default use, Pi for high-volume repeated jobs, Exo when retries are cheap, DSH for wall-clock speed.
More from coding & agent
- Claude Code + Monid MCP builds a $12K/year B2B prospecting engine on live data — aryanXmahajan · 2026-10-26
- Karpathy shares CLAUDE.md rulebook to stop AI coding agents from piling up tech debt — mdancho84 · 2026-10-26
- Rethinking dependencies in the agent era: drop libs that save 20 lines of code — gethackteam · 2026-10-26
- RustGIS hits 2,000 GitHub stars in two weeks with agent-first architecture — giswqs · 2026-10-26
- Vercel Ships 12 New Products at Its 2026 Conference as It Races Rivals for the Agent Entry Point — FinanceYF5 · 2026-10-26
- Build an AI agent from scratch in under 50 lines of pure Python: ReAct loop tutorial — adnan_hashmi · 2026-10-26