Geometric self-attention: 14,000x smaller state than KV cache with better CE at 2K context
iskander · x · 2026-10-15
Rohan Arun's team presents a geometric approach to self-attention: (1) function-weights represent weights as functions in GPU memory, letting a model sample new layers at inference from a curve (8 for phones, 24 for servers); (2) a hierarchical recurrent state replaces the KV cache with 14,000x less state and 50% better cross-entropy at 2K context, verified against official NanoGPT (11.64% lower CE in matched-parameter runs); (3) forward-pass perturbations between pretraining steps optimize the model without tokens; (4) experts are bundled and cached by use-case to compress GPU footprint. The goal is running frontier-style models on laptops; the team is also pushing a Lean integer multiplication bounds proof collaboratively on GitHub.
More from Infra
- Bloomberg: SpaceX AI ROIC could hit 25% by 2030, dwarfing CoreWeave and Nebius — elonmusk · 2026-10-15
- Nvidia's Ian Buck on Why GPUs Keep Winning: Blackwell Perf Up 4x Since Launch — firstadopter · 2026-10-15
- AI-rewritten libtiff closes gap to ~10x of C version, fueling hopes of retiring buggy C libs — jd_pressman · 2026-10-15
- PyTorch Day Japan 2026 agenda live: Tokyo event Dec 10, early-bird saves ¥3,000 — PyTorch · 2026-10-15
- Analyst asks: how many mechanics does a 500MW-to-GW data center need for manifold coolers — BenBajarin · 2026-10-15
- Cargo Demos 1.6T ALC Interconnect, ALC+AEC Cover More GPU Cluster Needs — BenBajarin · 2026-10-15