Anthropic's TASTE benchmark hides a shortcut; strongest models fall below chance

dhadfieldmenell · x · 2026-11-17

Researchers found a simple shortcut in Anthropic's TASTE benchmark (which probes LLMs' AI safety research taste). Once removed, accuracy anti-correlates with capability (r=-0.60, p<0.001) across 28 models: the weakest, Haiku 4.5, scores highest at 61.5%, while Anthropic's three strongest models hover near the 50% chance line and 8 of 9 OpenAI models score significantly below chance (GPT-6 Sol at 21.2%). The result casts doubt on whether top models genuinely possess safety research judgment.

Original post →

More from Models

Models channel →