Hamel Husain says evals beat vibes after Opus 5 costs 6× more and scores worse

HamelHusain · x · 2026-07-25

Hamel Husain argues that evals matter more than vibes after a quote about running Claude Opus 5 on a search-agent benchmark.

• The benchmark result says Opus 5 uses more than 6× the burn of 4.8 and more than 15× GPT-5.6, while performing worse than both.
• The accompanying transcript shows the model wavering between answers, changing its mind repeatedly before committing, illustrating unstable reasoning on a simple probability task.
• The post’s point is that benchmarked behavior should outweigh impressionistic praise when comparing frontier models.

Original post →

More from Models

• Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26
• AI search APIs ranked: top 12 within 7 points, yet prices differ nearly 7x — FinanceYF5 · 2026-10-26
• OpenAI Dots coming to EU/EEA and UK in a few weeks, slides confirm — testingcatalog · 2026-10-26
• OpenAI Dots demoed live as EU launch speculated, Codex adoption in Germany revealed — testingcatalog · 2026-10-26
• "Abliterated models just have brain damage" — community pushes back on unaligned model hype — dhadfieldmenell · 2026-10-26
• dair-ai publishes primer on Cloudflare's Clairic System One extraction model — dair_ai · 2026-10-26

Models channel →