Anthropic's TASTE benchmark hides a shortcut; strongest models fall below chance
dhadfieldmenell · x · 2026-11-17
Researchers found a simple shortcut in Anthropic's TASTE benchmark (which probes LLMs' AI safety research taste). Once removed, accuracy anti-correlates with capability (r=-0.60, p<0.001) across 28 models: the weakest, Haiku 4.5, scores highest at 61.5%, while Anthropic's three strongest models hover near the 50% chance line and 8 of 9 OpenAI models score significantly below chance (GPT-6 Sol at 21.2%). The result casts doubt on whether top models genuinely possess safety research judgment.
More from Models
- Pangram near-zero false positives on human text, but 'should you care?' debate rages — ipeirotis · 2026-11-17
- Gamma's 100M-user data: OpenAI leads efficiency, Anthropic nuanced understanding — omooretweets · 2026-11-17
- Google ships Gemini 4 Carbon in AI Mode but not in the Gemini app for Pro users — gaganghotra_ · 2026-11-17
- Ripple-65B Preview Debuts as US Open-Weight SOTA on Agentic Coding at 65B — Azaliamirh · 2026-11-17
- Tencent's hy-4 flash tested: the first Chinese model that's actually fast — ShunyuYao12 · 2026-11-17
- ChatGPT warns user over food chat from a year ago, safety backlash grows — Kyrannio · 2026-11-17