Inkling 2 Small adopts Kimi's attention residuals — mechanics explained in detail
akbirthko · x · 2026-11-13
Aleksa Gordić points out that Thinky Machines' newly open-sourced Inkling 2 Small adopts Kimi's attention residual mechanism, and breaks down how it works: each 6-layer block (6 attn + 6 MoE) keeps a running sum B of branch outputs; before every branch, a learned 4,608-dim query vector (shared across tokens) computes a softmax-weighted mixture over depth candidates [token embedding E, finished block sums, current B] after RMSNorm; branch outputs pass through a conv4 and are added only to the current block accumulator, with local causal channel-independent mixing; every 6 layers B becomes a new candidate and the accumulator resets, and a final learned query mixes 9 candidates at the end.
Related event: Thinking Machines Open-Sources Inkling 2 Small Weights under Apache 2.0(26 posts)→
More from Models
- Open-weight models are under 0.5% of T3 Code traffic — but that's customer data, not market data — Arindam_1729 · 2026-11-13
- Whisper vs Parakeet: people are fine-tuning NVIDIA's ASR model for medical use — MaziyarPanahi · 2026-11-13
- Pichai endorses Grok's model comparison: Hydrogen leads on price, Carbon on capability — sundarpichai · 2026-11-13
- Early tester builds a webcam air-sketching tool with Gemini 4 Hydrogen, impressed by its speed — CodeByPoonam · 2026-11-13
- Leaked screenshot shows Gemini's internal rules: ask before hallucinating on missing PDFs — Odd-Letterhead-6018 · 2026-11-13
- Musk touts Grok agent: sorts 2,000 Dropbox photos in five minutes — elonmusk · 2026-11-13