Garry Tan's 52-task eval: Fable 5.2 matches Opus 5.5 on pass rate but costs 2.80x more
garrytan · x · 2026-11-14
Garry Tan ran 52 Terminal-Bench 2.1 coding tasks in Claude Code 2.1.322 (2 tries each): Fable 5.2 solved 82.7% vs Opus 5.5's 84.6% — statistically indistinguishable — but at 2.80x the cost per solved task (CI 2.05x–3.99x).
Cost breakdown from the chart: Opus 5.5 solves tasks at about $0.65 each, Fable 5.2 at $1.83, and Fable 5.1 at $2.56 with only a 75% pass rate. His advice: keep Opus 5.5 as your Claude Code coding default; telling the two apart will require harder task sets.
More from coding & agent
- DeepSeek opens its entire app stack via API: search, voice, OCR, ASR, TTS — churchkey · 2026-11-14
- YC's Garry Tan open-sources agent harness evals: pass rates tie, costs differ 2.5x — garrytan · 2026-11-14
- T3 Code Now Lets You Fully Customize Your UI In-App; Agent Edits Update on the Fly — yihui_indie · 2026-11-14
- FieldKit: Offline Plugin Lets Hermes Agent Collect Forms, Photos and GPS Without Internet — Teknium · 2026-11-14
- Coinbase CEO: merged 46 PRs across 10 repos in 14 days part-time by talking to agents — ycombinator · 2026-11-14
- CoDelphi Reads Behavior Changes From Every Push So AI Tools Know the Blast Radius — tinyfool · 2026-11-14