LLNL releases SciResearch-Eval, a benchmark testing AI scientific judgment with broken simulators
alex_verem · x · 2026-11-19
The arXiv paper (2611.21541) behind LLNL's viral thread introduces SciResearch-Eval, the first interactive benchmark combining agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated 95% intervals. Across 93 held-out tasks, 9 models and 2,511 episodes: when the simulator is missing physics, most models report wrong values while claiming 95% confidence (coverage 20–62% vs 94% for GPT-6 Astra and 99% for Claude Fable 5.2). Stating the winning strategy improves interval scores 1.6–2.5-fold without moving coverage; only max-effort Claude 5.5 recovers, at 13–31x reasoning cost, while older and open-weight models gain nothing. Environment, graders and harness are released.
More from Research
- New LessWrong Analysis: LLMs Seek the Grader's Score, Not the Actual Goal — 1a3orn · 2026-11-19
- Multi Agent Coordination Bench Shows Agents Benefit From Randomness, Not Teamwork — soldni · 2026-11-19
- 90% of 533 agents herded into unethical behavior, and RL didn't fix it — karinanguyen · 2026-11-19
- FrontierRSI: New Benchmark Tests Whether AI Agents Can Do Real AI Research — techNmak · 2026-11-19
- Churchland lab builds RNN implementation of GLM-HMM to model engagement fluctuations — anne_churchland · 2026-11-19
- Energy-Based Reasoners: 1.8M-param model hits 99.9% on Sudoku-Extreme with 345x less inference compute — LucaAmb · 2026-11-19