GPQA Diamond flaw: question 186 requires an external telescope spec sheet link
charles_irl · x · 2026-10-21
After running GPQA Diamond, developer charlesirl found that question 186 includes a link to a European observatory telescope's spec sheet, which is needed to answer it — despite GPQA being framed as a non-agentic eval.
The model's reasoning trace shows it spiraling over the inaccessible link. His takeaway: "people are still not looking at the data," highlighting data-quality issues in a benchmark widely used to compare frontier models.
Related event: Flaw Found in GPQA Diamond Benchmark Question Requiring External Link(2 posts)→
More from Models
- Reflection's open-weight Beam 1.0 (501B-A23B) free on Nous Portal for a week — alexcovo_eth · 2026-10-21
- Karpathy: you know RL is working when models stop thinking in English — joeddav · 2026-10-21
- Devin Fusion's dual-model setup scores 68 on Coding Agent Index at 31% lower cost than Claude Code — ArtificialAnlys · 2026-10-21
- Perplexity open-sources pplx-decider models at 1/10th the cost of GPT 6 Luna — Balance- · 2026-10-21
- Ascend 950DT hits 1/5-1/7 of GB200 decode throughput on DeepSeek-V4.1, but EP optimization may close the gap — teortaxesTex · 2026-10-21
- Gemini 4 Carbon enters early access, spotted in LMArena, stronger at 3D than Argon — lyraxana · 2026-10-21