rasbt maps Inkling 2 Small's 594B architecture built on attention residuals
rasbt · x · 2026-11-14
Sebastian Raschka shared a hand-drawn architecture diagram of Inkling 2 Small, calling it more interesting than he expected.
Key points:
- 594B-parameter MoE: 8 shared router experts plus 48 router experts, with only 4 router experts active per token and 17B active parameters — no shared experts
- Uses block attention residuals (similar to Kimi K3) but without a linear-attention hybrid, combining short convs, GQA, and sliding window attention
- Supports up to 1M token context, 48 query heads, 8 KV heads, 200k vocab
- The diagram also compares Inkling 975B and Kimi K3.1 Pro (2.8T)
Raschka sees it as prime discussion material for an end-of-year architecture trends analysis.
Related event: Inkling 2 Small Architecture Draws Attention: 594B Params in Just 48 Layers(2 posts)→
More from Models
- Sokoban Eval: New 66-Room Benchmark Makes LLMs Play Classic Puzzle Game — banteg · 2026-11-14
- Lab post-training researchers flood-likes post mocking on-policy distillation reality gap — teortaxesTex · 2026-11-14
- Mitsuhiko: Anthropic has no image model, yet its code that paints is shockingly good — mitsuhiko · 2026-11-14
- Rapid Gemini releases at the frontier spark super-exponential AGI optimism — iruletheworldmo · 2026-11-14
- Google exec confirms Gemini 4 pre-training run has started, its most ambitious yet — Independent-Wind4462 · 2026-11-14
- Masked decoding tames Gemma 3 12B text artifacts with LoRA width tags and RL — round · 2026-11-14