Superposition is a geometric claim: near-orthogonal directions pack more features per layer
livgorton · x · 2026-10-16
In her thread, livgorton explains that superposition is fundamentally a claim about geometry: a layer represents more features than it has dimensions via almost-orthogonal directions, causing interference that is tolerable because features are sparse.
She notes this definition has remained consistent since Zoom In (2020), resting on features rarely co-occurring rather than on semantic unrelatedness.
More from Research
- SpliceAI2 benchmarks show major gains across nine splicing-variant evaluations — anshulkundaje · 2026-10-16
- GRPO post-training collapses model distributions, clashing with long-horizon agents — zsakib_ · 2026-10-16
- RFdiffusion3 update in Science shows major gains in protease and metallohydrolase design — anshulkundaje · 2026-10-16
- ORL pushes diffusion LMs to a new speed-intelligence Pareto frontier with ~6.9x throughput gains — james_y_zou · 2026-10-16
- Why CoT is the key: classic paper shows transformers need it for serial problems — willcb · 2026-10-16
- Circuit view may dissolve much of deep learning's mystery, argues researcher thread — willcb · 2026-10-16