Matryoshka Attribution tops interpretability benchmark by 2.9x, traces LLM refusals to 1% of weights
aryaman2020 · x · 2026-09-24
A new paper introduces Matryoshka Attribution (MAttr), a gradient-descent-based attribution method that finds which parts of a neural network drive a behavior.
- Ranks #1 on the Mechanistic Interpretability Benchmark by a wide margin (2.9x the runner-up)
- Frames classic interpretability problems as training objectives, directly optimizing attribution against the task of interest rather than axioms—answering the "Impossibility Theorems" call to action
- Uses a differentiable sigmoid top-k operator over causal interventions; randomizing the top-k budget during training yields a sparsity-agnostic attribution ranking, so eval can pick any k
- Applied result: using StrongREJECT as reward, refusals in Llama 3.1 8B Instruct are traced to 1% of the parameter delta from Base; resetting those weights largely disables refusal while keeping capabilities
More from Research
- Stanford's DyCraft Uses Agentic Decomposition to Cut Physical Errors in Long-Horizon 4D Scene Generation — StanfordUniversity · 2026-11-11
- Bert Chan on His Unconventional Path from Side-Projects to the ALife Institute — BertChakovsky · 2026-11-11
- CGM Study of 62,000 People: Average Glucose Beats HgbA1C at Detecting High Insulin — arjunrajlab · 2026-11-11
- First open dictation benchmark: 761 recordings across 10 apps, Wispr Flow 79% vs Apple 35% — rohanpaul_ai · 2026-11-11
- Boltz models showcased at MSK Cancer Center for accelerating early-stage drug discovery — charlieharris01 · 2026-11-11
- Carlman: a Pac-Man-style game where all characters are Lenia cellular-automata solitons — BertChakovsky · 2026-11-11