The Explainability Illusion: ViT patch heatmaps are a training artifact, not model property
ziv_ravid · x · 2026-10-20
A Technion paper by Schaffer, Manor and Michaeli shows that patch-token explainability in Vision Transformers — decoding individual patch tokens (à la Logit Lens) into human-interpretable heatmaps — is not an inherent property of the architecture. Whether a heatmap highlights the object, the background, or nothing at all depends on training data and sometimes even the random seed. Across architectures, scales and objectives, some ViTs show aligned token decodings while others show inverted maps or no semantic correlation at all. Theory backs this up: a single fixed toy transformer can provably implement the optimal classifier in multiple ways with different explainability patterns. The takeaway: you can't assume patch-decoding interpretability a priori — much of what's called interpretability may be reading tea leaves.
More from Research
- PretrainGym pits frontier agents on pretraining data curation: Kimi K3 wins at 1.042 — xeophon · 2026-10-20
- Claude runs 64-hour autonomous research: 318k protein designs, 17 binders validated — kimmonismus · 2026-10-20
- ACEdit Stabilizes 15-Turn Image Editing With Asymmetric Conditioning — tau · 2026-10-20
- Salesforce Trains Reasoning Models to Batch-Generate Diverse Research Ideas — Salesforce · 2026-10-20
- LightOn's blog details SlideOn's method, results and ablations — MoonL88537 · 2026-10-20
- SlideOn's sparse dims are vocabulary-independent, enabling inverted-index retrieval — tomaarsen · 2026-10-20