Garry Tan's open-source eval: 7 coding harnesses tie on capability but differ 2x on cost
garrytan · x · 2026-11-09
Y Combinator president Garry Tan shared early results from open-source eval work on coding harnesses with frontier models:
- Capability: can't tell yet. 7 setups (Claude Code, OpenClaw, Hermes, QM on Opus; Codex, OpenClaw, Hermes on Sol) ran the same 20 coding tasks, all passing 71–79%. Every comparison came out "can't tell apart" because 14 of 20 tasks scored identically everywhere — the test is too easy, a test problem rather than a tie. A harder Terminal-Bench rerun is underway.
- Cost and speed differ a lot. On Opus, OpenClaw pays $0.44 per solved task vs $0.22 for Claude Code, $0.24 for Hermes and $0.19 for QM; OpenClaw is also slowest at 99s per try vs 40s for Claude Code. On Sol, Codex is cheapest and fastest at $0.17. A second independent run confirmed the pattern.
- For now, choose on cost and speed: Claude Code on Opus, Codex on Sol.
Related event: YC President Garry Tan Open-Sources Benchmarks of 7 Coding Agent Frameworks(2 posts)→
More from coding & agent
- Relevant Isn't Authoritative: Why Agents Retrieve Right Docs and Still Decide Wrong — krishnan · 2026-11-09
- Two Agents Approve, One Missing Requirement: The Trap of Who Writes Acceptance Bars — krishnan · 2026-11-09
- Hands-on Hermes Agent review: model-neutral, runs GLM 5.3 and beats Codex for GPT — Scobleizer · 2026-11-09
- YC CEO Garry Tan open-sources his 'second brain' agent memory system, 31.7k GitHub stars — QuixiAI · 2026-11-09
- Free open-source AI engineering course makes you build backprop, attention and agent loops from raw math — ghumare64 · 2026-11-09
- Developer builds Shura: a local coding assistant on one RX 7800 XT hitting 78 tok/s that should improve with use — HIKKIAN · 2026-11-09