SO-Arena: NUS benchmark separates capability and decision reliability for single-pass models

新智元 · wechat · 2026-11-22

Researchers from NUS's xML Lab released SO-Arena, an open benchmark purpose-built for SystemOne models that output probability distributions in a single forward pass, spanning 2,919 tasks drawn from 40 public sources (MATH500, AIME, GPQA Diamond, Mind2Web, BFCL, etc.).

Four design principles

Key findings across 21 models

The takeaway for developers deploying SystemOne models: beyond accuracy, test whether models hold their judgment under option reordering and honestly abstain when no option fits.

Original post →

More from Models

Models channel →