LLNL releases SciResearch-Eval, a benchmark testing AI scientific judgment with broken simulators

alex_verem · x · 2026-11-19

The arXiv paper (2611.21541) behind LLNL's viral thread introduces SciResearch-Eval, the first interactive benchmark combining agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated 95% intervals. Across 93 held-out tasks, 9 models and 2,511 episodes: when the simulator is missing physics, most models report wrong values while claiming 95% confidence (coverage 20–62% vs 94% for GPT-6 Astra and 99% for Claude Fable 5.2). Stating the winning strategy improves interval scores 1.6–2.5-fold without moving coverage; only max-effort Claude 5.5 recovers, at 13–31x reasoning cost, while older and open-weight models gain nothing. Environment, graders and harness are released.

Original post →

More from Research

Research channel →