Artificial Analysis opens up coding agent leaderboard and benchmarking methodology
ArtificialAnlys · x · 2026-09-19
Artificial Analysis published the full results page and methodology for its coding agent benchmarks. Coding Agent Index v1.5 combines three equally weighted benchmarks: DeepSWE v1.1 (113 software engineering tasks, by Datacurve), Terminal-Bench 4.0 (66 agentic terminal tasks, by Laude Institute), and SWE-Atlas-QnA (124 technical Q&A tasks, by Scale AI).
Each benchmark averages pass@1 across three attempts per task. The leaderboard also tracks time per task and API cost per task, and notes that agents with similar index values can differ significantly across repository tasks, terminal workflows, and rubric-based evaluations.
Related event: Coding Agent Index Adds Safety Refusal Metric; Claude Tops at 8.8%(2 posts)→
More from coding & agent
- Why LLM-Written Docs Fail: Models Optimize for Punchiness, Not Readability — mervenoyann · 2026-11-04
- OpenAI DevDay demo shows forking agent task history at any step — paw_lean · 2026-11-04
- Veteran game dev: "AI builds a full game in a day" posts are mostly hype — bendee983 · 2026-11-04
- Dev builds interactive Excel-to-dashboard with ChatGPT, shares the prompt — TermRemarkable665 · 2026-11-04
- Agents cut dev time from weeks to a day; bottleneck moved to specs and review — Suspicious_Orchid770 · 2026-11-04
- Orgo, a YC-backed platform giving AI agents their own cloud desktops, claims 1,000+ teams — Scobleizer · 2026-11-04