Attention Head Stability Experiment Disproven
its_vayishu · x · 2026-07-16
The author tested whether attention heads maintain consistent "division of labor" across different inputs, examining not just if they resemble small-world networks, but if they actually perform the same specific functions.
Conclusions
- In the initial analysis, trained heads exhibited significantly more stable routing patterns with extreme statistical significance: p=2e-26.
- However, after excluding BOS / attention-sink tokens, the original difference disappeared, yielding p=0.32.
- This implies that the previously observed "structural sense" was likely an artifact caused by special tokens.
The author provided a complete notebook, all tests, and reproduction materials, encouraging readers to run it themselves.
Related event: Pitfalls in Analyzing GPT-2 Attention Maps(2 posts)→
More from Research
- Deep Learning Course Adds Mechanistic Interpretability Lecture This Winter — CSProfKGD · 2026-10-26
- INCLUDE-2.0 benchmark evaluates regional knowledge across 91 languages and 86 countries — ABosselut · 2026-10-26
- CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
- CoVer at EMNLP 2026: Training Code LLMs to Generate and Self-Verify Simultaneously — algo_diver · 2026-10-26
- Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
- Turing-complete malware can run in PyTorch's backward pass via a zero-valued loss — retr0jirachi · 2026-10-26