Coding Agent Index v1.5 adds safety refusal reporting; Claude Fable 5.1 fallback hits 8.8%
ArtificialAnlys · x · 2026-09-19
Artificial Analysis has added safety refusal reporting to its Coding Agent Index v1.5 to help explain score differences between models. A safety refusal occurs when a provider or model declines to start or continue a task on safety grounds, after which an agent may fall back to another model or stop with a block.
Claude Fable 5.1 showed the highest fallback rates in both Claude Code and Devin Fusion, with fallback attempts accounting for 8.8% and 7.1% of the Index's weight respectively — meaning those results partially reflect the fallback models' performance. The authors note that refusal variability, harness context buildup, effort settings, and retry strategies all affect observed rates.
Related event: Coding Agent Index Adds Safety Refusal Metric; Claude Tops at 8.8%(2 posts)→
More from coding & agent
- Claude Code adds AGENTS.md support in v2.1.277, falling back when no CLAUDE.md exists — BLUECOW009 · 2026-09-19
- Swarms Cloud adds full observability for every agent API execution — KyeGomezB · 2026-09-19
- Microsoft devs launch self-guided GitHub Copilot workshop: local repo to merged PR in 60-90 minutes — 0xkarasy · 2026-09-19
- Groovy's Updated AI Tutorial Covers Ollama4j, Spring AI, Embabel and Micronaut on JDK 25 — therealdanvega · 2026-09-19
- Jev, a typed-reasoning model that answers not writes, hits Venice API beta — 0xAllen_ · 2026-09-19
- Link launches agent wallet for AI shopping in Canada, with per-purchase approvals — jeff_weinstein · 2026-09-19