Spotlight architecture scrutiny: attention-style memory may carry a large constant factor
teortaxesTex · x · 2026-10-03
Citing Spotlight's pitch of combining attention's growing memory and capacity with the linear compute cost of fixed-state recurrent models, teortaxesTex questions the constant factor: a single memory access touches 9 cells, each cell a full dk×dv matrix per head. He suspects the overhead stems from continual learning design choices, offering a back-of-envelope challenge to the architecture's efficiency claims.
Related event: Percepta Unveils Spotlight: Decoupling Intelligence from Unbounded Memory(7 posts)→
More from Models
- Three Google flagship models in eight days fuel speculation about hidden RSI work — haider1 · 2026-11-19
- Mystery Model 'Gestalt' Tops New Frontier Design Benchmark by 214 Points — soleio · 2026-11-19
- mrfakename releases abliterated GLM-5.3-Flash and Qwen3.8-Flash-Next — mrfakename0 · 2026-11-19
- Physician: AI competition now measured in weeks, GPT-6 versions 7 days apart — ramos_casals · 2026-11-19
- micro1's flow qc model screens training data for benchmark leaks before release — Exp_Mark · 2026-11-19
- Leaker predicts new DeepSeek paper this week, scheduled for November 11 — teortaxesTex · 2026-11-19