Claude scored 0.26 on a science task, but the grader—not the model—may have failed
shoumikgoswami · reddit · 2026-11-18
Vedant Patel had Claude Opus 5.5 reproduce a palaeomagnetism analysis from raw data. The model estimated 32-34° of magnetic-field scatter vs. the paper's 37.5°, scoring 0.26. But rerunning with the paper's other published methods gave 32.3-37.5°, and against a range built from the authors' fits, the model's unchanged answers scored 0.50-0.73. Patel blames the grader for much of the apparent failure, though the model made real mistakes too, like keeping samples the authors flagged as bad. Lesson for eval builders: check whether your reference answer is the only acceptable one, and check intermediate work—a plausible final number can hide mistakes just as a strict grader can reject reasonable work.
More from Models
- LLM reviewers draw 50% of weaknesses from the appendix vs 8% for humans — plopesresearch · 2026-11-18
- Users ditching Fable 5.2 for Opus 5.5 as rate limits burn too fast — scaling01 · 2026-11-18
- Sol6.2 does PCB layout for ~$4 in 40 min, rivaling fable5.2 at 1/15 the cost — yacineMTB · 2026-11-18
- Qwen4-Max goes live on Runware with 1M-token context at $1.50 per 1M input tokens — Alibaba_Qwen · 2026-11-18
- GPT-6.2 tested on long-horizon finance: Sol regresses 2.4 points, Astra hits #2 — gabrielchua · 2026-11-18
- Qwen4-Max reportedly out; blogger tests it as an AI teammate on 3D and game dev — aftahi_ai · 2026-11-18