Released model is 1/3 of training size: pruned to 128 experts, healed with 1T distilled tokens plus 2.2T-token RL

nrehiew_ · x · 2026-11-21

nrehiew shares training details of the looped model: the released version is only about 1/3 of what was trained. Training started at sparsity 48 (8 of 384 experts), then two-thirds of experts were pruned mid-training, keeping the 128 most-used per layer to cut parameters for faster, local-hardware inference.

After pruning, the model trained on 1T tokens distilled from a larger internal Inkling model to recover pre-pruning performance, then went through SFT and an RL phase totaling 2.2T tokens over 2,000 steps. The author also teases a "larger internal Inkling."

Related event: Looping model loops only middle layers for 1.5-1.9x compute-equivalent gains(2 posts)→

Original post →

More from Models

Models channel →