Simulation scores, not vibes: when can you claim you've explained a neuron?
burny_tech · x · 2026-11-22
Responding to Nick Cammarata's argument that simulation scores are too demanding, the author counters that understanding comes in degrees — and simulation yields a deeper degree of understanding than correlation analysis or causal interventions like ablations and concept swaps.
The core claim: to claim you understand a neuron, it's insufficient for activating examples to look nice. You must be able to substitute simulated neuron values without degrading the model, using an explicit rubric ("dog literally on a bicycle scores 4.5; dog and bicycle but no riding scores 3.2"). If the substitution heavily degrades the model, the explanation was wrong.
The author concedes simulation KL isn't a perfect metric and is sympathetic to critiques of over-relying on metrics, but argues it avoids two failure modes that are hard to spot from activating examples alone.
More from Research
- LessWrong: looping boosts no-CoT reasoning far more than general skills in NVIDIA's Dory 4B — CFGeek · 2026-11-22
- "Hilbert's program vindicated": Lean brings us near a practical decision machine — teortaxesTex · 2026-11-22
- Model harnesses a year on: filesystem maximalism wins, long-lived named agents rise — voooooogel · 2026-11-22
- UChicago lab demos high-throughput precise perturbations, closing the loop toward autonomous live-cell control — arjunrajlab · 2026-11-22
- Weight-Space Editing Debate: One Researcher Calls It a Dead End, Another Bets on WSP + Phasor Error Correction — viksit · 2026-11-22
- Bytelisp: a bracket-free homoiconic Lisp for zero-data self-play pretraining — noahdgoodman · 2026-11-22