GPT-6-Astra scores 98% on unpublished theoretical physics problems, catching expert errors
sandersted · x · 2026-10-15
A large study (arXiv:2610.16733) led by physicist Suvrat Raju tested frontier AI agents on unpublished, expert-verified research-level theoretical physics problems.
- Extreme difficulty: organizers say no individual could solve a significant fraction of the final problem set without AI assistance; most problems they themselves couldn't even understand.
- Striking result: initial evaluation put GPT-6-Astra in the late eighties with Opus 5.5 close behind. When audited, Astra insisted the expert-supplied answers were wrong — and in multiple cases experts found subtle errors buried in their own calculations. After revisions, Astra finished at 98% (49/50).
- Its only failure wasn't a mistake: a trick prompt requiring an inspired guess and bending the rules, which the model can't do yet.
Raju said he "couldn't sleep properly for a couple of nights"; the paper warns of serious structural concerns around access, equity and pedagogy.
Related event: GPT-6-Astra Solves 98% of Unpublished Theoretical Physics Problems(3 posts)→
More from Models
- Claude Opus 5.5 tops SWE-bench Multimodal at 70.21%, nearly 9 points ahead — OfirPress · 2026-10-15
- Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error — RylanSchaeffer · 2026-10-15
- Test set contamination study rejected by NeurIPS: single exposure beats irreducible error floor — RylanSchaeffer · 2026-10-15
- New SWE-bench Multimodal leaderboard: Claude Opus 5.5 leads at 70.21% on visual bugs — OfirPress · 2026-10-15
- Google says Gemini 4 Argon is delivering breakthroughs in quantum computing, fusion and protein design — GoogleDeepMind · 2026-10-15
- Google sits down with Sergey Brin to discuss Gemini 4 Argon and the road ahead — OfficialLoganK · 2026-10-15