NUS launches SO-Arena: top System One model on accuracy ranks only 7th in reliability
NationalUniversityofSingapore · hf · 2026-11-13
NUS researchers released SO-Arena on Hugging Face, a benchmark covering 11 domains and 5 capability families to characterize System One models' capability spectrum and decision reliability. It features evidence-grounded evaluation with manually verified labels, fine-grained domain-specific difficulty criteria, realistic sandbox-based interactive evaluation, and reasoning-intensive tasks for System 1/2 comparison. Reliability is measured via decision stability under controlled option-set changes and missing-answer recognition. Evaluations across 21 models show scaling and reasoning gains vary substantially by domain, and strong overall accuracy does not ensure reliable decisions — the best-performing model ranks only 7th in reliability.
More from Research
- Tejas Kumar Publishes First Paper on Execution Contracts in Browser Agents — TejasKumar_ · 2026-11-13
- New RL textbook arrives; authors argue classic RL formulations remain untapped for LLM fine-tuning — AvivTamar1 · 2026-11-13
- Combining multi-agent objectives with single-agent action space breeds disempowerment — j_foerst · 2026-11-13
- Talk preview: one generative inference algorithm unifying Gestalt perceptual organization — neuroprinciples · 2026-11-13
- New research blog details training, eval, and use of decision models — antoine_chaffin · 2026-11-13
- NVIDIA's TTB bakes scene info into an MLP, speeding view synthesis 23x — nvidia · 2026-11-13