DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, Carolina Cuesta-Lázaro
stat.ML, cs.LG
2026-05-26
LLM agents must design experiments to recover non-canonical force laws in 22 simulated worlds. Claude Opus 4.7 reaches 50% pass@5; the best open model stays near 9%.
Most physics evals still test recall. GPQA-style graduate questions already sit above 90% for frontier models. Interactive suites such as NewtonBench, PhysGym, and Gravity-Bench get closer to experiment design, yet they mostly keep the law family and the named variables in view and only shift parameters.
Discovery often fails earlier: which variables matter, whether extra degrees of freedom are hiding, how to split signal from noise. DiscoverPhysics changes the force law itself, so an agent has to recover a mechanism from trajectories.
Twenty-two worlds are generated on demand by an N-body simulator. There is no static dataset. The force laws include Yukawa screening, fractional Laplacians, three hidden species, a dark-matter halo, ether drift, Hubble flow, time-varying coupling, and extra dimensions. Eleven worlds are public; eleven stay private against overfitting.
Each session gives the agent a fixed round budget. In a round it can place probe particles (position, velocity, charge, measurement times) and read noisy trajectories; the main results use position noise σ=0.05. It can also submit a candidate law and least-squares-fit up to five free parameters on data seen so far. When the budget ends it must hand in two artifacts: a short natural-language explanation and a Python force law that can be integrated.
A world counts as passed only if both bars clear: held-out normalized trajectory MSE below 0.1, and an explanation score of at least 0.9 against a human rubric, judged by claude-opus-4-6 (not in the reported baselines). A randomized-experiment mode samples probe initial conditions from a grid so the gap versus agent-chosen probes isolates experimental design from reading the data.
Eleven models, five seeds per world.
| Model | explanation | geo-mean MSE | pass@1 | pass@5 |
| claude-opus-4-7 | 0.61 | 0.01 | 26.4% | 50.0% |
| gpt-5.5 | 0.59 | 0.002 | 21.7% | 36.4% |
| Qwen3.5-397B | 0.33 | 0.1 | 4.5% | 9.1% |
| Llama-3.3-70B | 0.10 | 14.1 | 0 | 0 |
gpt-5.5 fits trajectories best and still trails Opus on pass rate and explanation. Raising the round budget from 2 to 16, Opus improves explanation and pass@3 monotonically; gpt-5.5 stops gaining explanation after about four rounds while MSE keeps falling. One agent revises the conceptual picture. The other locks a flexible equation early and tunes it.
Open models barely differ between chosen and random experiments, so the budget is mostly unused. Failures cluster on worlds with latent structure: three species, dark matter, extra dimensions. The oscillator world gives a clean counterexample. The same Opus, different seed: one run lengthens the measurement window and catches a sign-flipping G(t); the other stays on short timescales and files a 1/r law.
More noise usually hurts explanation. On the many-particle Ether world, Opus still flags the noise and redesigns probes at 50% noise. On two-particle Yukawa, explanation collapses past 20% noise.
This is currently the closest physics-discovery eval to "pick your own experiments, then revise the hypothesis." The two-axis score splits a good fit from a correct story. Reporting MSE alone would count curve fitting as discovery.
For people building science agents, the gap is experimental design, not Python. Open weights are not lab assistants on this task yet. pass@5 sits well above pass@1, so RL could farm the leaderboard; the authors warn that a higher score here would not imply transfer to real research.
The worlds are hand-written, not genuinely unknown physics. Particle counts are small (typically N ≤ 35) and the noise is Gaussian. A single Claude judge scores explanations; gpt-5.5 beating Opus at low round budgets argues against crude family bias, but a one-model judge remains a risk. The pass cutoffs (10% MSE, 0.9 explanation) are somewhat arbitrary, which is why mean scores are reported too. Growing the suite still needs humans to invent worlds and rubrics.