YC's Garry Tan open-sources agent harness evals: pass rates tie, costs differ 2.5x
garrytan · x · 2026-11-14
Y Combinator CEO Garry Tan open-sourced g-harness-evals, a benchmark that fixes the underlying model (Claude Opus 5.5, GPT-6.1 Sol) and tasks, then swaps only the agent harness (system prompt, tool calls, memory, permissions) to compare frameworks: Claude Code, Codex, OpenClaw, Hermes Agent, and QM, plus his own gbrain/gstack components.
First-round results on 20 dev tasks are counterintuitive:
- All 7 model+harness combos scored between 71% and 79% pass rate (Hermes 78.8%, Claude Code and OpenClaw 76.3%, QM 72.5%)
- 95% confidence intervals heavily overlap — statistically indistinguishable
- The real differentiator is cost per task, varying by more than 2.5x across harnesses
Takeaway: when picking a coding agent, don't fixate on the model — the harness layer moves cost far more than pass rate.
More from coding & agent
- DeepSeek opens its entire app stack via API: search, voice, OCR, ASR, TTS — churchkey · 2026-11-14
- Garry Tan's 52-task eval: Fable 5.2 matches Opus 5.5 on pass rate but costs 2.80x more — garrytan · 2026-11-14
- T3 Code Now Lets You Fully Customize Your UI In-App; Agent Edits Update on the Fly — yihui_indie · 2026-11-14
- FieldKit: Offline Plugin Lets Hermes Agent Collect Forms, Photos and GPS Without Internet — Teknium · 2026-11-14
- Coinbase CEO: merged 46 PRs across 10 repos in 14 days part-time by talking to agents — ycombinator · 2026-11-14
- CoDelphi Reads Behavior Changes From Every Push So AI Tools Know the Blast Radius — tinyfool · 2026-11-14