GPQA Diamond question 186 requires external telescope spec sheet, sending models spiraling
charles_irl · x · 2026-10-21
After running GPQA Diamond, charlesirl found question 186 (astrophysics) embeds a link to the ESPRESSO spectrograph spec sheet on ESO's site — data needed to answer — yet the benchmark is not agentic, so models can't actually fetch it.
The screenshot shows a qwen-3-8b reasoning trace (44k completion tokens, 663s latency) spiraling over 100k characters about how to estimate ETC throughput parameters, even debating deriving zero points from first principles or reconstructing ETC from ESO docs.
It highlights a benchmark design flaw: closed-ended scoring on questions that depend on external data inflates hallucination and reasoning loops.
Related event: Flaw Found in GPQA Diamond Benchmark Question Requiring External Link(2 posts)→
More from Models
- Musk: Grok Bot is the fastest-growing product he's ever seen, enterprise rollout in weeks — elonmusk · 2026-10-21
- Google: Gemini 4 Argon processed ~25 trillion tokens in first 72 hours — OfficialLoganK · 2026-10-21
- Leaked GPT-6.1-Sol reasoning traces show familiar GPT caveman CoT once spaces are restored — teortaxesTex · 2026-10-21
- Gemini 4 Scores Just 78% on FrontierMath Tier 4 as OpenAI Models Saturate It — haider1 · 2026-10-21
- Hamel Husain: public benchmark quality is abysmal — broken graders, sloppy harnesses — HamelHusain · 2026-10-21
- 163MB open-source speech model Bark 3.0 beats Whisper (57x bigger) and NVIDIA Parakeet (15x bigger) — Scobleizer · 2026-10-21