Hamel Husain says evals beat vibes after Opus 5 costs 6× more and scores worse
HamelHusain · x · 2026-07-25
Hamel Husain argues that evals matter more than vibes after a quote about running Claude Opus 5 on a search-agent benchmark.
- The benchmark result says Opus 5 uses more than 6× the burn of 4.8 and more than 15× GPT-5.6, while performing worse than both.
- The accompanying transcript shows the model wavering between answers, changing its mind repeatedly before committing, illustrating unstable reasoning on a simple probability task.
- The post’s point is that benchmarked behavior should outweigh impressionistic praise when comparing frontier models.
More from Models
- ParseBench puts Claude Opus 5 near Opus 4.8 on docs, but cheaper rivals win on tables — llama_index · 2026-07-25
- Claude Opus 5 is shown hitting 449.46× speedup on a kernel benchmark — scaling01 · 2026-07-25
- Claim says Claude Opus 5 scored 42/42 on the 2026 International Math Olympiad — Polymarket · 2026-07-25
- Claude Opus 5 feels better at turning messy inputs into a sharp insight, user says — iruletheworldmo · 2026-07-25
- Claude Opus 5 gets treated like it has “taste” in a meme-style repost — repligate · 2026-07-25
- Claude Opus 5 is being shared for a quirky “back to work” behavior clip — repligate · 2026-07-25