SO-Arena: NUS benchmark separates capability and decision reliability for single-pass models
新智元 · wechat · 2026-11-22
Researchers from NUS's xML Lab released SO-Arena, an open benchmark purpose-built for SystemOne models that output probability distributions in a single forward pass, spanning 2,919 tasks drawn from 40 public sources (MATH500, AIME, GPQA Diamond, Mind2Web, BFCL, etc.).
Four design principles
- Verifiable labels: human checks that each reference answer is supported by model-visible evidence;
- Difficulty tiers: 9 of 11 domains split into difficulty levels to preserve discrimination;
- Interactive evaluation: 619 environment tasks run models through multi-step web/tool sandboxes, scoring full task success;
- Decision reliability: an independent reliability score R combining option-shuffle stability (S) and null-option recognition (N — noticing when the correct answer is missing), reported alongside capability score C.
Key findings across 21 models
- GPT-6 Sol (82.3) and Luna (79.7) still lead; the best SystemOne models GLiDE and KnowLine-31B score 70.2, but run under 0.3s per task on a single H100 — dozens of times faster than closed APIs;
- Gaps to SystemTwo vary by domain: semantic judgments are nearly at parity; fine-grained decisions benefit more from bigger backbones than from thinking; multi-step math lags most, where enabling thinking lifts StartLux Auto from 50.8 to 92.5;
- Capability and reliability rankings diverge: the most reliable model (JevAny-27B, R=0.84) ranks only 11th on capability; shuffle stability roughly tracks capability, but null-option recognition varies widely even at similar capability — some 4B models beat 27B ones, so it must be measured directly.
The takeaway for developers deploying SystemOne models: beyond accuracy, test whether models hold their judgment under option reordering and honestly abstain when no option fits.
More from Models
- Andrew Ng: Gemini Neon is the only model that doesn't talk like an insufferable AI — andrewgwils · 2026-11-22
- Reflection AI lets its Beam agent run a real vending machine to probe autonomy risk — alexpolozov · 2026-11-22
- Bindu Reddy: Neon high matches Opus high quality but runs much slower — bindureddy · 2026-11-22
- Claude Max/Ultra users get $100–$200/mo in API credits most never claim — here's how to actually use them — jekapats · 2026-11-22
- TensorFold 1.3.3 ships 124 PRs in 24h, boosts GLM decode to 172 tok/s on GB10s — AIFlow_ML · 2026-11-22
- OpenAI Sweeps Top 7 Spots on Updated Arena Alignment Index; Kimi K3.1 Pro Best Open Model — gdb · 2026-11-22