New Decision Index benchmarks: decision models match LLMs on classification at 1/4 the cost
ArtificialAnlys · x · 2026-10-14
Artificial Analysis launched the Decision Index v1.0, evaluating decision models across four benchmarks (AA-Omniscience-Decision, HLE-MultiChoice, When2Call, WildGuardTest). Key results:
- Jev 1.13 (TypeSafe) leads decision models at 58, ahead of GLiDE and Microsoft-Decision-1 (both 54);
- Perplexity's Decider v1.1 27B is cheapest at $17 per 1M tasks; Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also sit on the cost-capability Pareto frontier; Celeris-1 Decision is fastest at 0.06s per task;
- Decision models rival LLMs on classification and tool calling at far lower cost/latency, but trail on knowledge and reasoning: Claude Haiku 5.5 (Low) scores 61 but is 11x slower and 4.4x more expensive than Jev 1.13;
- Most decision models are poorly calibrated on knowledge/reasoning tasks, better calibrated on classification and tool calling.
More from Models
- Gemini 4 takes the crown on GamowLabs' RareBench and LabBench, dethroning Opus 5.5 by a healthy margin — sahandsharif · 2026-10-14
- Muse Wonder's under-discussed edge: first frontier lab to report 'big model smell' anti-benchmaxxing metric — alexandr_wang · 2026-10-14
- TV noise crashes voice agents: task completion falls from 38.7% to 8.6% across 14 stacks — solyarisoftware · 2026-10-14
- GPT-5.6 Sol finds critical Chrome vulnerability after 6 days and $40,000 of compute — gdb · 2026-10-14
- Early Muse Wonder impressions: no personality regressions, subtle bug catches and better UI sense — alexandr_wang · 2026-10-14
- Pro users frustrated as Gemini 4 Argon skips Antigravity access — Wonderful-Season-173 · 2026-10-14