Simulation scores, not vibes: when can you claim you've explained a neuron?

burny_tech · x · 2026-11-22

Responding to Nick Cammarata's argument that simulation scores are too demanding, the author counters that understanding comes in degrees — and simulation yields a deeper degree of understanding than correlation analysis or causal interventions like ablations and concept swaps.

The core claim: to claim you understand a neuron, it's insufficient for activating examples to look nice. You must be able to substitute simulated neuron values without degrading the model, using an explicit rubric ("dog literally on a bicycle scores 4.5; dog and bicycle but no riding scores 3.2"). If the substitution heavily degrades the model, the explanation was wrong.

The author concedes simulation KL isn't a perfect metric and is sympathetic to critiques of over-relying on metrics, but argues it avoids two failure modes that are hard to spot from activating examples alone.

Original post →

More from Research

Research channel →