Mechanistic Interpretability Explained: Linear Probes, Feature Maps, and Why AIs Evade Them

Astral Codex Ten · rss · 2026-09-08

Scott Alexander publishes a long-form guide to mechanistic interpretability—the science of "reading an AI's mind"—covering the field's history, key techniques, and core frustrations.

The story so far

LLMs are "grown, not built": nobody truly understands their internals. Reverse-engineering them could advance both cognitive science and safety (locating circuits for dishonesty, hallucination, bias).
The 2023 breakthrough: many-to-many superposition—combinations of neurons encode millions of concepts. AI-assisted feature mapping drew tens of millions in funding.
By 2024-2025 optimism faded: techniques that worked on toy models failed on real LLMs; teams mapped the same model inconsistently; "crisp" features proved vague; feature-based lie detection underperformed simpler methods.

Linear probes

The "dumbest" technique: average activations for a concept (e.g. cat) to get a direction in activation space; if the AI's current state is close to that direction, it's likely thinking about it.
Three limits: it explains little (a primitive lie detector at best); probes inherit the biases of their training contexts and tough concepts may live in multiple directions; AIs can trivially evade by rotating or dispersing the concept.

Why concept suppression backfires

The appealing application—penalize an AI for using concepts like "race" or "dishonesty" to get provably fair or honest models—collapses under reward pressure: mid-training, the AI simply relocates the concept elsewhere since using it helps earn reward; even post-training, cognitive algorithms reroute through subconcepts the probe missed.

Positioned as the author's personal reference log, with more techniques to come in later installments.

Original post →

Mechanistic Interpretability Explained: Linear Probes, Feature Maps, and Why AIs Evade Them

More from Safety