Possible eval leak: all frontier models fail a GPQA Diamond question whose official answer seems wrong
qberthet · x · 2026-11-12
gbarthmaron noticed that all frontier models get a certain GPQA Diamond astrophysics problem wrong — but after checking, he believes the official target answer itself is incorrect. The question: two stars (1.5 and 1.2 solar masses) with abundances measured via the EW method under LTE ([Si/Fe]1 = 0.3 dex, [Mg/Si]2 = 0.3 dex, [Fe/H]1 = 0, [Mg/H]2 = 0; solar photosphere 12+log10(nFe/nH)=7.5, 12+log10(nMg/nH)=7), asking for the ratio of silicon atoms between the two photospheres. qberthet shares it asking astrophysicists to weigh in: mass model failure, or a mislabeled benchmark item?
More from Models
- ICLR abstract word-frequency shift: 'novel' and 'comprehensive' plummet as LLM fingerprints fade — y_m_asano · 2026-11-12
- Codex quota reset caveat: using a reset pushes your next refresh 7 days out — JeremyNguyenPhD · 2026-11-12
- Gemini 4 Carbon hands-on: beats DeepSeek Flash, priced at Sonnet level not Opus — oran_ge · 2026-11-12
- Doubao Work adds third-party models; DeepSeek V4.1 Flash edits an 8-min video for $0.02 — op7418 · 2026-11-12
- LLMs play Among Us against each other; Haiku 5.5 kills then instantly self-reports — Rasmic · 2026-11-12
- Kimi K3.1-Pro abliterated variant hits Hugging Face, claiming 95% refusal removal — zebird0 · 2026-11-12