Kimi team says delta-rule state space models are more expressive than attention
teortaxesTex · x · 2026-07-26
Kimi team argues that state space models with the delta rule are strictly more expressive than attention, challenging the idea that attention is all you need.
The post also suggests this may be easier to train end to end than the current cascade of sparsity tricks used in V4, and hopes both approaches can work well.
Related event: Kimi K3 Architectural Innovations Spark DRAM Demand Debate(5 posts)→
More from Research
- MIT's Ziming Liu: AutoResearch doesn't generalize — blind training tricks cause 'forking' disasters — ZimingLiu11 · 2026-10-26
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26