Coding Agent Index adds refusal views across DeepSWE, Terminal-Bench, SWE-Atlas-QnA
ArtificialAnlys · x · 2026-10-02
Companion post to Artificial Analysis's refusal-reporting launch, linking the full methodology: the Coding Agent Index v1.5 equally weights DeepSWE v1.1 (113 tasks), Terminal-Bench 4.0 (66 tasks), and SWE-Atlas-QnA (124 tasks), averaging pass@1 over three attempts per task across 31 models. Refusal timing and fallback views are available per benchmark, alongside time- and cost-per-task metrics.
Related event: Artificial Analysis Adds Safety Refusal Analysis to Coding Agent Index(2 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16