Mech interp is a data problem: 10K labeled tasks could localize reward hacking

burny_tech · x · 2026-11-08

@sensho argues mechanistic interpretability can be treated as a data engineering problem: build 10K tasks — 5K clearly defined as reward hacking, 5K guaranteed clean — run rollouts, and measure activation differences to identify clusters that predict reward hacking. The bottleneck is the tedious data work of creating such tasks, not an algorithmic breakthrough.

Original post →

Mech interp is a data problem: 10K labeled tasks could localize reward hacking

More from AGI Musings