Inkling 2 Small quietly ships: 48-layer model with attention residuals and new sinks
PMinervini · x · 2026-11-14
liuliu notes Inkling 2 Small has been downloadable since yesterday with little discussion, and breaks down its architecture:
- Attention Residuals: at 48 layers (vs Hy4's 78), attnres spans 8 blocks of 6 layers each;
- New attention sink / gated attention: beyond the known sigmoid gated attention, the new sink uses per-head projected value (per-token per-head) rather than a per-head constant;
- It carries over Inkling's relative biases and short conv (canon layer) designs.
He urges the community to study it as closely as they analyze DeepSeek's Sparse Attention and other architectural tricks.
Related event: Inkling 2 Small Architecture Draws Attention: 594B Params in Just 48 Layers(2 posts)→
More from Models
- rasbt maps Inkling 2 Small's 594B architecture built on attention residuals — rasbt · 2026-11-14
- Sokoban Eval: New 66-Room Benchmark Makes LLMs Play Classic Puzzle Game — banteg · 2026-11-14
- Lab post-training researchers flood-likes post mocking on-policy distillation reality gap — teortaxesTex · 2026-11-14
- Mitsuhiko: Anthropic has no image model, yet its code that paints is shockingly good — mitsuhiko · 2026-11-14
- Rapid Gemini releases at the frontier spark super-exponential AGI optimism — iruletheworldmo · 2026-11-14
- Google exec confirms Gemini 4 pre-training run has started, its most ambitious yet — Independent-Wind4462 · 2026-11-14