New leaderboard ranks 57 decision models: d1-ultra tops browser tasks at 74% for $0.026 per 1K decisions
AIMultiple · reddit · 2026-10-21
AIMultiple launched the AIM System One Index, a leaderboard for 57 'System One' decision models that return typed decisions with probabilities in a single pass instead of generating tokens. It combines three benchmarks: 50 one-shot browser tasks, 1,655 single-decision classification questions (security advisory severity, FDA recall classes, etc.), and bias tests with a rotated 'cannot be determined' option. Scores are z-scores (50 = average, 10 = one SD); cost and latency are shown but excluded. AutoJev-27B, d1-ultra and decider-12b tie at the top with 60. d1-ultra leads browser tasks at 74% and costs $0.026 per 1,000 decisions; GLiDE hits 72% but needs 15.7s per decision; Laya is fastest at 30ms; Decider Flash V1.2 is cheapest at $0.007 per 1,000 decisions.
More from Research
- Recursive training on its own safe outputs makes models progressively misaligned, warns study — burny_tech · 2026-10-21
- CPLM: sliding-window Transformer matches full attention with 12x less KV cache — burny_tech · 2026-10-21
- SPACE: undo part of SFT with zero retraining to boost LLM scores both in-domain and out-of-domain — kaiwei_chang · 2026-10-21
- One table shows speculative decode and agent loops are the same algorithm at different layers — akbirthko · 2026-10-21
- Ampersand's Persimmon user model cuts models' evaluation awareness by 40 points — alexisjross · 2026-10-21
- Snorkel AI audit: 55% of TerminalBench 4.0 fixes had reward hacking issues themselves — ravithejads · 2026-10-21