Sonnet 5.5 effort levels tested: max scores 55.5 at $134.79 vs 9 for Sonnet 5
PawelHuryn · x · 2026-09-29
Pawel Huryn benchmarked all Sonnet 5.5 effort levels on his real-repo bug-fixing benchmark (n=2 averages):
- max: 55.5, 1,330 turns, 234.7 min, $134.79
- xhigh: 39, 588 turns, $60.30
- high: 32, 294 turns, $16.73
- medium: 18, 112 turns, $8.39
- low: 20, 119 turns, $5.75
Sonnet 5 (max) scored just 9 at $17.96. Overall: an incredibly motivated model — fast and cheap per turn, but it runs far more turns when needed, with max costing 25x+ the low setting.
Methodology: real bugs re-planted in real repos that frontier models missed in early 2026, blind calibrated judges from a different family, only truly fixed bugs count, answer key kept private.
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16