Researcher rebuts superposition skeptics: near-orthogonal capacity, interference and recursive generalization
livgorton · x · 2026-10-16
Liv Gorton threads a point-by-point response to an essay challenging superposition:
- Superposition is a geometric claim: a layer represents more features than dimensions via near-orthogonal directions; sparsity makes interference tolerable. A d-dim space has only d orthogonal but exponentially many near-orthogonal directions, so networks exploiting this isn't "radical".
- Anticorrelated features are cheap to superpose; semantic relatedness isn't the point — related but mutually exclusive features interfere little and still get superposed.
- Correlated features can share space too, preferring positive over negative interference; recent work finds constructive interference shaping clusters and cycles in real models.
- Recursive generalization (a neuron growing from "iy" to "do it yourself") is a learning-dynamics claim, not an alternative; but distinguishing nested features downstream still needs some mechanism, and superposition is the simplest candidate. The essay's own readout story "sounds a lot like superposition".
More from Research
- SpliceAI2 benchmarks show major gains across nine splicing-variant evaluations — anshulkundaje · 2026-10-16
- GRPO post-training collapses model distributions, clashing with long-horizon agents — zsakib_ · 2026-10-16
- RFdiffusion3 update in Science shows major gains in protease and metallohydrolase design — anshulkundaje · 2026-10-16
- ORL pushes diffusion LMs to a new speed-intelligence Pareto frontier with ~6.9x throughput gains — james_y_zou · 2026-10-16
- Why CoT is the key: classic paper shows transformers need it for serial problems — willcb · 2026-10-16
- Circuit view may dissolve much of deep learning's mystery, argues researcher thread — willcb · 2026-10-16