Attribution graph evidence questioned: SAEs systematically bias toward shattered 'circuits'
banburismus_ · x · 2026-10-18
@banburismus argues that if attribution graphs are the primary evidence that circuits get harder with scale, new work suggests this is a systematic problem with SAEs: the method has consistently biased researchers against finding structure and toward shattered, bag-of-features "circuits," while looking differently (e.g., manifold representations) tends to reveal more structure.
The linked arXiv paper, "Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds," provides evidence: the authors implement three block-sparse featurizer (BSF) variants and show via minimum-description-length analysis that all describe activations more compactly than direction-based featurizers, with recovered concepts typically 2-4 dimensional low-dimensional manifolds—echoing the classical neuroscience idea that visual features are carried by coordinated groups of neurons rather than single tuned ones.
Related event: Study Finds LLMs Use Fourier Features for Addition with Just 28 Neurons(2 posts)→
More from Research
- Hexagon, a new math preprint repository that accepts works arXiv won't, launches in beta — tak3sh8 · 2026-10-18
- Periodic's CoLM talk: when does training your own model beat frontier + harness? — heghbalz · 2026-10-18
- PNAS paper shows biological synapses can self-assemble learning rules akin to gradient descent — xuanalogue · 2026-10-18
- SGPnP paper accepted at IEEE TCI: diffusion priors plug into PnP reconstruction with theory — prof_kamilov · 2026-10-18
- Surya Ganguli lecture traces the statistical-physics origins of scaling laws — burny_tech · 2026-10-18
- Domingos: superposition is a feature, not a bug — it boosts memory, not blocks interpretability — pmddomingos · 2026-10-18