Opus 5.5 Benchmarks Can't Be Verified Outside Anthropic — Author Proposes Cheap Independent Audit
maier_ak · x · 2026-10-02
Andreas Maier's essay argues hosted model quality can be set administratively by the seller via controls buyers can't observe. He traces Anthropic's release cadence: Opus 4.7 (Apr 16) drew a record 35 unwarranted-refusal reports in a month, 4.8 followed 42 days later; Opus 5 (Jul 24) drew complaints of verbose answers and massive unnecessary rewrites, with 5.5 arriving 60 days later. His point isn't fraud but that the launch page answers questions nobody outside the company can check. He proposes a cheap audit: pinned snapshots, stated effort per benchmark column, and a held-out suite run daily by someone not selling the model.
Key points
- All launch-page numbers are self-measured, third-party irreproducible
- 5.5's price cuts (input $5→$4, output $25→$20, cache reads -60%) undercut the 'cripple the old' theory
- But the claimed 40% serving-cost reduction implies savings come from curtailed deliberation
- Calls for an industry-wide cheap independent verification mechanism
Related event: Opus 5.5 Benchmarks Questioned as Vendor Grades Its Own Homework(3 posts)→
More from Models
- Mystery Model 'Gestalt' Tops New Frontier Design Benchmark by 214 Points — soleio · 2026-11-19
- Contralabs Launches Frontier Design Benchmark, Claims Designers Still Beat AI — yuwen_lu_ · 2026-11-19
- Physician: AI competition now measured in weeks, GPT-6 versions 7 days apart — ramos_casals · 2026-11-19
- micro1's flow qc model screens training data for benchmark leaks before release — Exp_Mark · 2026-11-19
- Leaker predicts new DeepSeek paper this week, scheduled for November 11 — teortaxesTex · 2026-11-19
- Gemini 4 Neon rollout begins: leaked UI shows 3-tier Gemini 4 lineup with Carbon locked — testingcatalog · 2026-11-19