LLMs hit gold-medal level on astronomy olympiad, Gemini 2.5 Pro scores 85.6%
bravo_abad · x · 2026-08-21
A new paper benchmarks five frontier LLMs on exams from the International Olympiad on Astronomy and Astrophysics — far harder than typical scientific QA, requiring deep conceptual understanding, multistep math derivations, and multimodal analysis, the capabilities that matter if LLMs are to act as scientific agents rather than retrieval systems.
Across four theory exams from 2022–2025, Gemini 2.5 Pro scored 85.6% and GPT-5 84.2%, both in gold-medal territory among 200–300 human participants; on data-analysis exams GPT-5 reached 88.5%, comparable to top-10 competitors. Yet the authors argue the error analysis is more informative than the scores: solving exams is still not doing science, and the models' mistakes show why.
More from Models
- Kimi K3.1 Pro tops offensive cyber eval at 46.43%, claims of deliberate nerfing surface — teortaxesTex · 2026-10-25
- OpenAI and Anthropic reportedly blocked REA tool reverse engineering privacy-invading smart TVs — kroggens · 2026-10-25
- Ternary weights shrink a speech model from 2,472 MB to 181 MB with just +0.4pt WER — lmoroney · 2026-10-25
- Unverified DeepSeek-V4.1-Flash appears on Hugging Face with 763B params, llama.cpp PR filed — jacek2023 · 2026-10-25
- DeepSeek-V4.1-Flash EXL3-3bpw quant runs on 2x RTX PRO 6000 — No-Paper-557 · 2026-10-25
- Dev argues Codex still beats Opus 5.5 at coding: Opus brute-forces bad designs instead of researching — haider1 · 2026-10-25