Audit finds TASTE safety benchmark shortcut: fewer interp terms beats all LLMs

aryaman2020 · x · 2026-11-17

An audit of TASTE — a benchmark of 92 pairs of AI safety research proposals where experts pick the better one — found a shortcut: mech-interp-themed proposals (probes etc.) tend to be worse. Simply picking proposals with fewer interp terms achieved higher accuracy than all tested LLMs, suggesting benchmark scores are contaminated by surface lexical features.

Original post →

More from Research

Research channel →