SSOG Attention: Sub-Quadratic Alternative to SDPA with Separable Gaussians, Beats on CIFAR-100
4rtemi5 · reddit · 2026-08-16
Introducing SSOG (Sum Of Separable Gaussians) attention, a sub-quadratic alternative to scaled dot-product attention (SDPA). SDPA has O(N²·d) complexity, while SSOG learns a few Gaussian atoms per head and steers them based on query tokens, achieving O(N·√N·d) via separable factorization. Experiments show SSOG clearly outperforms SDPA on CIFAR-100 and matches performance with faster convergence on ImageNet-1k, while being more memory-efficient at scale. Blog post and code are open-sourced.
More from Research
- Retriever: A Framework for Asynchronous, Closed-Loop Robot Agents — ZeYanjie · 2026-08-24
- Converting GMMs ↔ PEFs for fast KLD approximation — FrnkNlsn · 2026-08-24
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24