Kimi Delta Attention is framed as a regression rule mapping queries to keys
burny_tech · x · 2026-07-28
The post points readers to a video explaining Kimi Delta Attention and linear attention more broadly. The core takeaway is that the differences between Mamba2, Gated DeltaNet, and Kimi Delta Attention may be smaller than they look: the key question is how the state update rule is derived.
The update is described as a cumulative sum of the outer product q kᵀ, and in DeltaNet terms it is likened to an SGD step with MSE loss. In other words, the intuition is that these methods can be seen as training a regression model that maps queries to keys.
More from Research
- MIT's Neil Thompson: AI comes for tasks before jobs, and fixed costs favor big firms — bigdata · 2026-10-16
- UMD's PAIR framework bridges perception and action in VLA models via bridge tokens — Kaixi Feng · 2026-10-16
- CI-MSE offline metric hits -0.87 Spearman correlation with real robot task success vs weak raw MSE — chris_j_paxton · 2026-10-16
- Colossal Biosciences uses AI to decode genetic switches behind the extinct dodo — castrotech · 2026-10-16
- Colossal Biosciences uses AI to identify gene switches in dodo de-extinction push — castrotech · 2026-10-16
- Cheng Lou open-sources Handwired: hand-written weights turn code into training-free neural nets — dotey · 2026-10-16