Audit finds TASTE safety benchmark shortcut: fewer interp terms beats all LLMs
aryaman2020 · x · 2026-11-17
An audit of TASTE — a benchmark of 92 pairs of AI safety research proposals where experts pick the better one — found a shortcut: mech-interp-themed proposals (probes etc.) tend to be worse. Simply picking proposals with fewer interp terms achieved higher accuracy than all tested LLMs, suggesting benchmark scores are contaminated by surface lexical features.
More from Research
- Winning team details agentic protein design with Lumos for Adaptyv x Anthropic contest — CatAstro_Piyush · 2026-11-17
- Together Deleted Its Megakernels: AI Agents Tune Inference 28% Faster on B200 — togethercompute · 2026-11-17
- Self-improvement tricks for multimodal agents fail to transfer across scenarios, 44-page survey finds — rohanpaul_ai · 2026-11-17
- Agentic robotics brings simulation back to center stage for embodied AI — Lianhuiq · 2026-11-17
- NVIDIA open-sources cuPotential, 3x faster molecular dynamics on Blackwell GPUs — AllThingsApx · 2026-11-17
- Modeling the Unitree G1's closed-loop push-rod ankle in MuJoCo — kevin_zakka · 2026-11-17