Arena Introduces Factuality Leaderboard
Kyrannio · x · 2026-07-17
Arena has introduced a new model ranking dimension: factuality.
- Rankings are no longer based solely on human preference; instead, human preference and factuality are combined into a weighted metric. This new view is available as a non-default toggle in Text Arena and Search Arena.
- Their approach involves randomly sampling battles, extracting web-verifiable claims from model responses, verifying each claim for accuracy, and comparing the average accuracy across different models.
- To support this leaderboard, Arena has annotated over 2 million LLM claims, including 1.3 million+ from Text Arena and 700,000+ from Search Arena.
- A related video explains how this method extends the Bradley-Terry objective to account for both preference and factuality, and why models shouldn't be penalized simply for "refusing to answer / making no assertions."
Related event: Arena Adds Factuality to Model Rankings(12 posts)→
More from Research
- rauchg reproduces OpenAI's covert CPU-channel finding: chat across Firecracker microVMs with no network — ccerrato147 · 2026-10-27
- Microsoft Research: ForkPilot cuts long-horizon agent compute by up to 59% — EliasEskin · 2026-10-27
- Microsoft's TeleTune evolves agent skills from offline user telemetry — EliasEskin · 2026-10-27
- FlexGlue: sparse-to-dense matching that beats dense accuracy at a fraction of the cost — zhenjun_zhao · 2026-10-27
- Mundo turns a few room photos into an explorable, simulatable 3D world — at 1.5 FPS — mark_k · 2026-10-27
- Mathematics as an experimental science: Marc Lelarge to talk LLM-driven math experiments at ENS — marc_lelarge · 2026-10-27