Hamel Husain says evals beat vibes after Opus 5 costs 6× more and scores worse

HamelHusain · x · 2026-07-25

Hamel Husain argues that evals matter more than vibes after a quote about running Claude Opus 5 on a search-agent benchmark.

• The benchmark result says Opus 5 uses more than 6× the burn of 4.8 and more than 15× GPT-5.6, while performing worse than both.
• The accompanying transcript shows the model wavering between answers, changing its mind repeatedly before committing, illustrating unstable reasoning on a simple probability task.
• The post’s point is that benchmarked behavior should outweigh impressionistic praise when comparing frontier models.

Original post →

More from Models

• Bug Hunt Bench ranks frontier coding models on 105 planted real-repo bugs — PawelHuryn · 2026-09-11
• GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
• 105 hidden bugs, 2 repos: DeepSeek V4.1 Flash fixes 24 at $1.80 vs Opus 5's 27 at $51.33 — ChartsJournalX · 2026-09-11
• awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
• Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
• Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11

Models channel →