Neel Nanda's team launches WorkspaceBench, an eval to test interpretability tools
burny_tech · x · 2026-09-24
- ML researcher Neel Nanda announced WorkspaceBench, an eval for interpretability tools: a set of scenarios where researchers know what the model should be thinking, testing whether your interp method can surface the important intermediate variables (the "global workspace") in a forward pass.
- Motivation: progress in ML is bottlenecked by good evals, and interpretability is no exception.
- The quoted post notes that "Astra" can do impressive things without chain-of-thought — which is scary, and interpretability is the remedy.
- The author adds that J-Lens is strong but only outputs a single token; a good multi-token J-Lens should do well on WorkspaceBench.
More from Safety
- AI Defense Shifts to Scale Economics: Attacks Buried in Legitimate Traffic — DavidLinthicum · 2026-11-11
- AI safety researcher hires paid, credited RA/co-writer to speed up memos and blog posts — davidmanheim · 2026-11-11
- How long should an approval for an agent action stay valid? — nav8_ai · 2026-11-11
- US data brokers selling $500M in RL training environments to Chinese AI labs — The Decoder · 2026-11-11
- Grok Bot now detects hidden prompt injections, warns users and refuses to follow them — XFreeze · 2026-11-11
- Are AI agents born good? Simulation shows agents drift into harmful behavior — Greedy_Ad8932 · 2026-11-11