Loop the Loopies!
Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai
cs.CL, cs.AI
2026-07-17
Loopie rewrites looped-Transformer recurrence as layer-loop (layer1 to layer1 to layer2 to layer2), overtaking a vanilla compute-matched baseline and matching a 30B model on one-seventh the pretraining tokens, reaching 92 on AIME 2024.
Looped Transformers try to save parameters by running the same set of layers N times: one network executes repeatedly, trading parameters for effective compute. They carry a chronic handicap: running N times multiplies pretraining compute by N, so the fair opponent is a model with roughly N times the parameters, not a same-parameter one. Historically looping lost that fight. Given N times the compute, scaling parameters by N usually beats looping the model N times. Loopie targets exactly this "looping penalty."
The key change is the granularity of recurrence. Prior model-loop runs "layer1 then layer2 then layer3 then layer1 then layer2 then layer3," repeating the whole network. Loopie uses layer-loop: each layer runs twice before moving on, "layer1 then layer1 then layer2 then layer2 then layer3 then layer3."
Two models: Loopie-20B-A2B (27 layers, hidden 2304, 128 experts with 8 active per token, 2 loops) and Loopie-6B-A0.6B (18 layers, hidden 1536, same 128 experts and 8 active, 2 loops). The recipe halves the stored layers, executes each twice, then reinvests the freed memory into doubling per-device microbatch and ploughing the efficiency back into capacity. Recurrent compute is not free; under a matched budget you either train on fewer tokens, use a smaller stored architecture, or compare against a stronger non-recurrent model. The paper repeats this honest caveat.
For reasoning, a new post-training stage called SPT (Supervised Pre-Training) combines SFT's supervision with pretraining's optimization scale, processing 128 million tokens per global batch, roughly 1000 times conventional SFT. The pipeline is pretraining, high-quality annealing, SPT (2T tokens), math RL, then Loopie Thinking.
Against a compute-matched vanilla 30B-A3B (both trained on 800B tokens), Loopie lags early and overtakes after about 600B tokens. Across a four-rung scaling ladder (0.15B to 1B active parameters), Loopie beats the compute-matched vanilla baseline at every rung, and the gap does not vanish with scale. A loop-count sweep finds recurrence most beneficial at R=2, so the whole series uses 2 loops.
Token efficiency is strong: Loopie pretrains on 3.5T tokens versus 25T each for Nemotron 3 Nano and Nemotron Cascade 2, less than one seventh, yet matches or exceeds them on most knowledge and general benchmarks. Loopie-20B-A2B Thinking scores 92.09 on AIME 2024, 83.75 on AIME 2025, 81.28 on MMLU; the 6B version reaches 80.42 on AIME 2024.
The looped-Transformer line had been largely written off as "loses to parameter scaling." Loopie is the first work to systematically reverse that verdict under matched compute. For anyone focused on training efficiency, it shows that a correctly designed loop (layer-level, not whole-network) can buy comparable capability with far fewer pretraining tokens. That matters most for compute-constrained teams.
Post-training covers only math and code reasoning; scientific QA and conversational alignment are untouched. SPT, the new post-training method, lacks a thorough ablation because compute was limited. There is no systematic study of inference-time compute. The architecture deliberately stays on one clean base (Qwen3-30B-A3B) without testing recent architectural advances. In the "matches 30B" comparison, Nemotron uses 25T tokens and Loopie 3.5T; the token count lines up, but data mix and post-training pipeline may not be comparable, and the paper does not fully isolate how much of the gap comes from the loop structure itself.