A ladder of interpretability evidence: from correlation to full mechanism simulation
burny_tech · x · 2026-10-25
A framework for grading the strength of evidence in mechanistic interpretability:
- Correlation: a feature/neuron lights up on "cat" inputs
- Steering: adding the direction pushes outputs toward "cat"
- Ablation: removing it breaks cat-related behavior
- Counterfactual concept swapping: patching "dog" activations where "cat" was flips downstream behavior coherently
- Simulation: you can rebuild the mechanism well enough to predict or reproduce the model's behavior
The author also flags advanced pitfalls: interventions on one mechanism may indirectly intervene on another that's the real target, requiring proper localization; confounder analyses across the space of confounders; "self-repairing" mechanisms after intervention; and ensuring the intervention itself doesn't create the mechanism being studied.
More from Research
- Molecular Cell review maps emerging frontiers of AI in single-cell genomics — anshulkundaje · 2026-10-25
- Harvard's Zitnik lab unveils CONCERT, a generative model for spatial perturbation responses — marinkazitnik · 2026-10-25
- Sander Dieleman's 84-minute deep dive: flow maps learn the integral of diffusion models to speed up sampling — sedielem · 2026-10-25
- AI science loops have too much textbook, not enough Newton-style curiosity — generativist · 2026-10-25
- Stanford's Finn lab proposes Generative Value Classifier, recasting Q-learning via action likelihoods — chelseabfinn · 2026-10-25
- Economist: skeptics were right about AI growth, bottlenecks still misunderstood — Afinetheorem · 2026-10-25