Coding Agent Index v1.5 adds safety refusal reporting; Claude Fable 5.1 fallback hits 8.8%
ArtificialAnlys · x · 2026-09-19
Artificial Analysis has added safety refusal reporting to its Coding Agent Index v1.5 to help explain score differences between models. A safety refusal occurs when a provider or model declines to start or continue a task on safety grounds, after which an agent may fall back to another model or stop with a block.
Claude Fable 5.1 showed the highest fallback rates in both Claude Code and Devin Fusion, with fallback attempts accounting for 8.8% and 7.1% of the Index's weight respectively — meaning those results partially reflect the fallback models' performance. The authors note that refusal variability, harness context buildup, effort settings, and retry strategies all affect observed rates.
Related event: Coding Agent Index Adds Safety Refusal Metric; Claude Tops at 8.8%(2 posts)→
More from coding & agent
- Hindsight ships 0.8B model for agent memory decisions: 2.5x cheaper, 1.9x faster — victor_explore · 2026-11-04
- Human Oversight Is 70-75% of AI Agent Costs in Banking, McKinsey Finds — mikeflache · 2026-11-04
- Over 80% of Major Financial Institutions Now Run Agentic AI in Production — mikeflache · 2026-11-04
- Mnemos MCP nears launch after shared memory layer fix — RileyRalmuto · 2026-11-04
- Moving mobile dev to remote machines makes it multiplayer across product, design, engineering — rudrank · 2026-11-04
- Kuaishou's EvoRec: self-evolving agentic recommender lifts ad revenue 4.6% — _reachsumit · 2026-11-04