Artificial Analysis Launches Intelligence Index v5 with 60% Private Data; Opus 5.5 Tops as Open-Weight Gap Widens to 12 Points

Artificial Analysis released Intelligence Index v5 on October 28, alongside the new Progress Index v1. This is the biggest overhaul since v4 launched in January: four capability dimensions each weighted 25%, a total of 9 (reportedly 12) benchmarks, and 60% of the weight drawn from private test sets or held-out answers to combat data contamination. Claude Opus 5.5 tops the overall ranking, while the open-weights camp's deficit has grown from 3 points in v4.3 to 12 points.

Confirmed

• Weighting structure: Knowledge Work (25%, including AA-Briefcase 15%, AutomationBench-AA, etc.) plus three other dimensions each at a quarter; new benchmarks include AA-Briefcase v1.1, AutomationBench-AA, Terminal-Bench 4.0, and FrontierCode 2.0, with HLE-Diamond replacing Humanity's Last Exam
• Heavier evaluations: median output tokens per task jumped from 35k to 59k; effort levels matter more—Claude Opus 5.5 scores 35 at low effort vs. 58 at max effort, with per-task costs differing 10x ($0.9x vs. $10.04)
• Cost landscape: Opus 5.5 (max) costs $10.04 per task, GPT-6 Astra (max) $6.24, while GPT-6.1 Sol (max) costs just $1.27 with a near-comparable score; a Pareto chart relayed by @BorisMPower shows OpenAI dominating the "intelligence per dollar" frontier at nearly every price point, with the priciest apex as the sole exception
• Open-weights falling behind: leader Kimi K3.1 Pro (weights unreleased) scores 46, 12 points behind the top; among released-weights models, the DeepSeek V4 series performs best but still trails by 18 points on FrontierCode 2.0
• No single model sweeps: Opus 5.5 leads on only 3 benchmarks, including FrontierCode 2.0 (46%) and HLE-Diamond (68%); the other 6 are split among 5 different models, and all nine evaluations leave ample room for improvement
• Sub-leaderboards: on CritPt v2's private subset (40 research-grade physics problems), GPT-6 Astra leads at 72% vs. an open-source best of just 45%; in HLE-Diamond, knowledge questions trip up frontier models more than reasoning ones; GPT-6 Astra (max) tops Terminal-Bench-Science at 63%, with most open-source models under 10%; FrontierCode 2.0, newly released by Cognition, features 132 real open-source repo tasks scored by whether maintainers would merge the change
• Progress Index v1: places all models since November 2022 on a single capability scale, enabling year-over-year comparison

Why It Matters

• The 60% private/held-out data weighting directly addresses benchmark contamination, a persistent industry pain point, likely boosting the leaderboard's credibility—though full external reproduction becomes harder
• Effort levels are now a key scoring variable for the first time: same-model score gaps can reach 23 points with 10x cost differences, demanding more nuanced reading of both "model scores" and "value for money"
• The open- vs. closed-source gap has widened from 3 to 12 points, with even starker disparities on agentic benchmarks like terminal science and code maintenance—a wake-up call for the open-source camp

2026-10-28 ~ 2026-10-28 · 15 related posts

• Artificial Analysis unveils Intelligence Index v5 with four 25% categories and 12 benchmarks — xeophon · 2026-10-28
• [source] Artificial Analysis Intelligence Index v5: 60% private datasets, open-weights gap widens to 12 points — ArtificialAnlys · 2026-10-28
• FrontierCode 2.0: Claude Opus 5.5 tops Cognition's coding benchmark at 46% — ArtificialAnlys · 2026-10-28
• Terminal-Bench-Science: GPT-6 Astra 63%, most open-weights models under 10% — ArtificialAnlys · 2026-10-28
• HLE-Diamond: Claude Opus 5.5 leads at 68% as knowledge questions stump frontier models — ArtificialAnlys · 2026-10-28
• CritPt v2 hard subset: GPT-6 Astra leads at 72%, top open-weights model only 45% — ArtificialAnlys · 2026-10-28
• No model sweeps AA v5: Opus 5.5 leads three of nine evaluations — ArtificialAnlys · 2026-10-28
• Open weights fall 12 points behind in AA v5, gap triples from v4.3 — ArtificialAnlys · 2026-10-28
• GPT-6.1 Sol matches Opus 5.5 within 4 points at one-eighth the cost per task — ArtificialAnlys · 2026-10-28
• Effort settings now swing scores 20+ points at 10x cost, AA v5 finds — ArtificialAnlys · 2026-10-28
• AA v5 tasks consume 59k median output tokens, up two-thirds from v4.3 — ArtificialAnlys · 2026-10-28
• [source] Artificial Analysis launches Progress Index v1 to track capability since Nov 2022 — ArtificialAnlys · 2026-10-28
• [source] Artificial Analysis Intelligence Index v5: nine benchmarks across four equally weighted pillars — ArtificialAnlys · 2026-10-28
• Artificial Analysis releases Intelligence Index v5: Claude Opus 5.5 leads at 58 — socoolandawesome · 2026-10-28
• Artificial Analysis v5 index: OpenAI dominates every price-performance point except the priciest — BorisMPower · 2026-10-28