Artificial Analysis launches Decision Index v2 with 45% private data and calibration-based scoring
ArtificialAnlys · x · 2026-11-09
Artificial Analysis released Decision Index v2, rebalanced to resist benchmark gaming: private datasets now make up 45% of the index (up from 25%). The new headline Decision Score rewards both correctness and calibrated confidence, equivalent to 50 × (1 + Accuracy − Brier score).
New components:
- AA-Paperclip (30%): 1,500 private documents (resumes, receipts, invoices, papers), each with 5 decisions + 10 fact extractions; a document only passes if all 15 answers are right
- AA-Symmetry (15%): 120 base questions and 440 variations testing consistency of answers and probabilities
- New EvalSafe-ONET-AA (15%); When2Call down to 10%; WildGuardTest removed for saturation
Key results:
- Top decision models within 1 point: Drex Max v1.1 (Nace AI) and GLiDE v1.2 (Fastino) at 73, Mammoth 27B (Base Labs) at 72; Claude Haiku 5.5 (Max) scores 77, GPT-6 Luna (Max) 76
- Cost per 1M tasks spans 60x+: from $14 (Celeris-1.1) to $859 (Clef); GPT-6 Luna (Max) costs $1,530 vs. $102 for Drex Max v1.1
- Billing structure matters: GLiDE v1.2 bills document input per question, making it cost $780 — nearly 8x Drex Max v1.1 despite only 33% higher list prices
Related event: Artificial Analysis Launches Decision Index v2(2 posts)→
More from Models
- Model recipe via rigorous experiments, but scaling could rewrite the story — liliyu_lili · 2026-11-09
- GLM-4.5 Hit #1 on Hugging Face Within 48 Hours of MIT-Licensed Release — ccerrato147 · 2026-11-09
- Has any LLM release ever been decidedly worse than its predecessor? — adonis_singh · 2026-11-09
- CEOs say Google's Argon stands out for writing, calling Google the AI dark horse — danshipper · 2026-11-09
- Epoch AI adds ECI scores, projects Gemini 3 Pro time horizon at 4.9 hours — CFGeek · 2026-11-09
- Infosec researcher maps 3 low-hanging abuse paths for open-weight models — matthew_d_green · 2026-11-09