SSOG Attention: Sub-Quadratic Alternative to SDPA with Separable Gaussians, Beats on CIFAR-100
4rtemi5 · reddit · 2026-08-16
Introducing SSOG (Sum Of Separable Gaussians) attention, a sub-quadratic alternative to scaled dot-product attention (SDPA). SDPA has O(N²·d) complexity, while SSOG learns a few Gaussian atoms per head and steers them based on query tokens, achieving O(N·√N·d) via separable factorization. Experiments show SSOG clearly outperforms SDPA on CIFAR-100 and matches performance with faster convergence on ImageNet-1k, while being more memory-efficient at scale. Blog post and code are open-sourced.
More from Research
- GAE paper learns a geometry-native latent space for 3D-consistent world generation — yshan2u · 2026-10-16
- Agents turn conservative as context grows: edits shrink from 150 to 30 lines — dbreunig · 2026-10-16
- LMR-BENCH at EMNLP 2025 tests whether LLM agents can reproduce real LLM research — ZhiyuChen4 · 2026-10-16
- Ex-OpenAI researcher claims most neurons are legible now; critics call it hype — anshulkundaje · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Survey: AI agent evaluation is getting realistic but still fails to measure safety and reliability — burkov · 2026-10-16