MaD Physics benchmark rewrites physical laws to test AI; only one model noticed

weballergy · x · 2026-10-29

MaD Physics (arXiv:2605.10820) is a new benchmark evaluating scientific agents' ability to make informative measurements under budget constraints and infer underlying physical laws, filling a gap left by benchmarks that only test static knowledge or unconstrained experimental design.
It includes three environments with deliberately altered physical laws to mitigate knowledge contamination: agents must plan measurements within a budget, then infer the law and predict future system behavior.
A third-party account describes results claiming only GPT-6 Astra noticed anomalies in the altered-physics world and experimentally deduced its unique laws (heavier objects fall slower, red spheres attract, periodically fluctuating gravity), while 11 other models (GPT-6.1 Sol, Grok 4.7, Kimi K3, etc.) never did; Astra also proactively investigated anomalies, unlike Gemini 3.7 Flash which continued its planned experiment.
In the normal-physics world, about a third of AI reports spuriously claimed the laws had changed.
Note: model names and test narrative come from third-party retelling; consult the paper for authoritative details.

More from Models