SO-Arena Benchmark: Top-Scoring Model Ranks Only 7th in Decision Reliability
kalyan_kpl · x · 2026-11-15
A new paper introduces SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families.
Benchmark features
- Evidence-grounded evaluation with manually verified reference labels
- Fine-grained capability assessment via domain-specific difficulty criteria
- Realistic interactive evaluation
Findings across 21 models
- Benefits of scaling and extra reasoning vary substantially across domains
- Strong overall performance does not imply reliable decision-making: the best-performing model ranks only seventh in reliability
More from Research
- Stanford's Flash ALOHA robot arms catch a thrown ball in 1.85s with just 50 demos — k7agar · 2026-11-15
- Drone swarms tie knots in midair to lift objects autonomously — lukas_m_ziegler · 2026-11-15
- Fleuret: v1.4 training was a disaster, but FCA-style models hit 41.17% on GSM8K — francoisfleuret · 2026-11-15
- World Model Hallucination Paper Wins Best Paper at CoRL 2026 PhysWM Workshop — LerrelPinto · 2026-11-15
- RNN memory is set by recurrent matrix eigenvalues — LRU's ring init fixes it — RobertTLange · 2026-11-15
- Ropedia Xperience 2: aligning scene scans and human actions for robot simulation reuse — liuziwei7 · 2026-11-15