DeltaNet notes unpack Kimi Delta Attention with equations and a state-update diagram
nrehiew_ · x · 2026-07-22
A technical thread breaks down DeltaNet / Gated DeltaNet / Kimi Delta Attention from first principles, using equations and a diagram to show how the state update works.
- It starts from linear attention by removing softmax and rearranging matrix operations.
- It then introduces the delta rule, where the recurrent state accumulates new association information while correcting duplicated old associations.
- Gated Delta Net adds data-dependent gates (alpha, beta) to decay the previous state, erase old value associations, and write new ones.
- The image compares the update rules side by side and highlights how Kimi Delta Attention scales each dimension with a data-dependent diagonal matrix before applying the delta-style update.
- The post is framed as study notes from a conversation and points readers to the original thread for a fuller walkthrough.
More from Companies & People
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27
- Anthropic reveals it bans customers worth 5-7% of revenue every 9 months — Polymarket · 2026-10-27
- Anthropic's IPO Filing Goes Public, Revealing Strong Revenue and Massive Spending — KateClarkTweets · 2026-10-27
- Microsoft offers voluntary buyouts to 60,000 employees, about a quarter of its workforce — dotey · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Anthropic IPO timing fuels claims anti-open-weights panic is self-serving — mark_k · 2026-10-27