Inkling 2 Small architecture detailed: attention residuals, NVFP4/MXFP8 mixed-precision experts
teortaxesTex · x · 2026-11-13
Inkling 2 Small's architecture is getting a close read. Yang Songlin highlights attention residuals (with q/k/v split), gated attention, dynamic attention sink, dynamic shortconv and value norm. Elie Bakouch adds: experts use NVFP4 in layers 0-39 and MXFP8 in the last 8, softcaps in the lm head and SwiGLU, and sliding-window attention bumped from 512 to 1k. Per the diagram: MoE at every layer (384 experts, top-8, 12B params each), 40 sliding-window layers plus 8 global attention layers with sink logits and 128K context, a 201,024-token vocab with tanh softcap 30, and depth reads over block sums.
More from Models
- AI Daily: Huang resigns, Anthropic IPO at up to $1.4T, Gemini 4 Carbon opens — xiaohu · 2026-11-13
- Dylan Patel: Moonshot and DeepSeek's open-source code only targets Blackwell — anshulkundaje · 2026-11-13
- Tesla FSD handles a dark, snowy Finnish road and yields to a bus smoothly — yunta_tsai · 2026-11-13
- Claude now refuses to work on open-source projects that ban AI contributions — esrtweet · 2026-11-13
- Thinking Machines' Inkling 2 Small (594B MoE, 1M context) lands on Ollama — ollama · 2026-11-13
- No forensic evidence DeepSeek's V4.1-Flash used Blackwell, buried in report appendix — suchenzang · 2026-11-13