Claude scored 0.26 on a science task, but the grader—not the model—may have failed

shoumikgoswami · reddit · 2026-11-18

Vedant Patel had Claude Opus 5.5 reproduce a palaeomagnetism analysis from raw data. The model estimated 32-34° of magnetic-field scatter vs. the paper's 37.5°, scoring 0.26. But rerunning with the paper's other published methods gave 32.3-37.5°, and against a range built from the authors' fits, the model's unchanged answers scored 0.50-0.73. Patel blames the grader for much of the apparent failure, though the model made real mistakes too, like keeping samples the authors flagged as bad. Lesson for eval builders: check whether your reference answer is the only acceptable one, and check intermediate work—a plausible final number can hide mistakes just as a strict grader can reject reasonable work.

Original post →

More from Models

Models channel →