Artificial Analysis launches Coding Agent Index v2.0: Claude Opus 5.5 leads at 61.7
ArtificialAnlys · x · 2026-10-23
Artificial Analysis released Coding Agent Index v2.0, evaluating model-plus-harness combos across 290 tasks spanning software engineering, terminal use, and repo Q&A with three equally weighted benchmarks.
What changed
- FrontierCode v1.1 (from Cognition) replaces DeepSWE: 100 tasks on real open-source repos with a private test set, graded on whether a maintainer would merge the change; attempts flagged for accessing solution material score zero.
- SWE-Atlas-QnA grading revised to -AA: same 124 tasks, 127 of 1,306 rubric criteria reworded, judge switched to GPT-6 Luna (xhigh) with five votes per criterion; dataset and judge config published on Hugging Face.
Key results
- Claude Code + Claude Opus 5.5 (max) leads at 61.7, ahead of Sonnet 5.5 (59.3) and Muse Code + Muse Wonder Full (58.9).
- On FrontierCode, Opus 5.5 scores 56.0% vs Codex + DeepSeek V4.1 Flash at 52.6%; 15 of 24 configs cluster between 43–49%.
- Stricter grading: mean score across 21 overlapping configs fell 1.7 points, with drops up to 7 points.
- Cost frontier: DeepSeek V4.1 Flash at $0.39/task (47.3), GPT-6.1 Sol $1.50 (54.1), GPT-6 Astra $7.04 (54.7), Opus 5.5 $11.42 (61.7), all pay-per-token API costs.
More from coding & agent
- NVIDIA unveils OpenShell Reference Architecture to set boundaries for enterprise AI agents — NicoVerderosa · 2026-10-23
- Personal AI agents become a capital-intensive race as cloud execution environments reshape cost structures — sujingshen · 2026-10-23
- LLMs make code so cheap that recursive rewrites could make verified software tractable — jd_pressman · 2026-10-23
- Hugging Face hands OpenEnv RL environments project to Linux Foundation for neutral governance — lmoroney · 2026-10-23
- Theo writes the definitive explainer on what an AI coding "harness" actually is — pvncher · 2026-10-23
- Conga launches MCP Server letting Claude and Copilot query live contract data — shashib · 2026-10-23