Released model is 1/3 of training size: pruned to 128 experts, healed with 1T distilled tokens plus 2.2T-token RL
nrehiew_ · x · 2026-11-21
nrehiew shares training details of the looped model: the released version is only about 1/3 of what was trained. Training started at sparsity 48 (8 of 384 experts), then two-thirds of experts were pruned mid-training, keeping the 128 most-used per layer to cut parameters for faster, local-hardware inference.
After pruning, the model trained on 1T tokens distilled from a larger internal Inkling model to recover pre-pruning performance, then went through SFT and an RL phase totaling 2.2T tokens over 2,000 steps. The author also teases a "larger internal Inkling."
More from Models
- Gemini 4 Neon locked to Pro and Ultra paid subscribers, leaving Plus users out — giffmana · 2026-11-21
- Looped transformers: more looping boosts reasoning but with diminishing returns past 3 passes and fewer CoT tokens — nrehiew_ · 2026-11-21
- Team releases TML-d24-varloop-research on Hugging Face to study looping's effect on CoT monitorability — nrehiew_ · 2026-11-21
- Full sub-benchmark breakdown for Inkling 2 Mini published by Artificial Analysis — ArtificialAnlys · 2026-11-21
- Inkling 2 Mini costs ~6x less and runs ~4x faster per task than Qwen3.8 27B, AA finds — ArtificialAnlys · 2026-11-21
- Inkling 2 Mini matches 27B-33B rivals on intelligence with ~5-6x fewer active parameters — ArtificialAnlys · 2026-11-21