Interpretability researcher lists top open problems in decoding model activations

wesg52 · x · 2026-09-16

JackWLindsey outlines what he sees as the most important current research questions, centered on "mind-reading" model activations.

Multiple activation-decoding techniques now exist, but all have clear failure modes: NLAs are often hallucinatory, while Jacobian-lens-style methods only support bag-of-words readouts and capture just part of the activation vector.
The paradigm of decoding single-token, single-layer activations may be inherently limited — models use a whole context's worth of activations, so decoding entire contexts may be the right target.
He also calls for better characterization of simple baselines like "just ask the model what it's thinking about."

He argues progress on these failure modes is fairly tractable.

More from AGI Musings