PINNACLE: Claude Fable 5.1 halves agent failure rate but costs $2.46 per correct answer
ryanshrout · x · 2026-09-03
Two days after Signal65 launched its PINNACLE leaderboard, Anthropic shipped Claude Fable 5.1, and the numbers show why it exists:
- Failure rate: agent workflow failures drop from 14/100 (where Claude Opus 5 and GPT-5.6 Sol sit) to 7/100 — roughly half
- Hallucination: it invents an answer on 0.7% of unanswerable questions vs 7.6% for Opus 5 (1 in 140 vs 1 in 13)
- Why it matters: that gap moves agents from "a person checks everything" to "a person checks the exceptions"
- The catch: $2.46 per correct answer, 1.8x Opus 5, mostly from context the agent re-reads every step; the bill scales with volume with no knob to turn it down
Fewer failures or a smaller bill, per role, is the trade-off PINNACLE is built to surface.
Related event: Claude Fable 5.1 Halves Agent Failure Rate in PINNACLE Benchmark(2 posts)→
More from Infra
- SGLang Summit sponsors grow from 11 to 36 in one week; Lip-Bu Tan, Lilian Weng to speak — BanghuaZ · 2026-10-18
- Yacine jokes his datacenter KPI is gigawatt-hours: get paged if not burning enough power — yacineMTB · 2026-10-18
- OpenAI reportedly handled ~15T tokens daily in April, likely higher now — JoshPurtell · 2026-10-18
- 1.68x inference speedup drops, directly accelerating GRPO post-training — yacineMTB · 2026-10-18
- OpenRouter is only a fraction of the total addressable market, poster argues — sloppenheimer · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18