Attention as spherical interpolation: values combine inside the unit sphere before norm projects back
mgostIH · x · 2026-11-06
mgostIH offers a geometric intuition for attention: the attention operation is a convex combination of values, which for unit-sphere vectors almost always lies inside the sphere. Layer/RMSnorm are projections onto the sphere, so attention+norm acts as spherical interpolation. Since high-dimensional Gaussians are nearly spherical, this frames Transformer internals as movements on high-dimensional spheres. An accompanying diagram illustrates the convex combination inside the unit circle and projection back to the surface.
More from Research
- Study: coding agents follow stale CLAUDE.md 90% of the time—a one-sentence fix boosts code compliance to 91.4% — petrusenko_max · 2026-11-06
- 1,400-line teaching agent harness scores full marks on Terminal Bench 2.1 — PawelHuryn · 2026-11-06
- Dev releases 1.4M-image pixel-art synthetic dataset on Hugging Face under CC0 — Angaisb_ · 2026-11-06
- Hugging Face publishes a three-part deep dive into Mixture of Experts — ariG23498 · 2026-11-06
- ETH Zurich's ECHO implant converts excess blood sugar into urine-excretable gluconate — CurieuxExplorer · 2026-11-06
- Science: cracks emerge in Lean certificates backing AI-generated math proofs — neurovium · 2026-11-06