Loopie Looping Transformers Match Larger Models at Fraction of Cost

Zitian Gao's team released the Loopie series of looping Transformers, aiming to improve both parameter and compute efficiency by repeatedly executing network layers. The series includes two MoE versions: 20B total parameters (2B active) and 6B total parameters (0.6B active). This research has drawn significant attention because it not only overcomes the historical scaling bottlenecks of looping architectures but also claims to achieve large-scale model performance at extremely low computational costs.

Key Details and Breakthroughs

In the past, increasing pretraining compute for looping Transformers was often less effective than simply scaling up parameters. First author Zitian Gao stated that the team spent hundreds of millions of dollars in compute to validate the classic Universal Transformer approach (the model-loop scheme), finding it scaled poorly to large-scale MoE architectures with billions of parameters. Later in pretraining, the team's proposed Layer-loop scheme (the Loopie model) demonstrated superior scalability, successfully overcoming previous bottlenecks. According to @Dany0's estimate, the training cost for this kind of model might only be in the range of hundreds of thousands to a million USD.

Performance and Evaluation Controversies

In terms of performance, the paper claims that the 20B model beats a same-compute 30B baseline under the same training wall-clock time and a matched inference compute budget. Even more notably, @Dany0 pointed out that the model used only 10% of the pretraining tokens to match or even exceed the performance of Qwen3 Coder 30B. Regarding this research, @teortaxesTex cautioned the industry that evaluating looping Transformers cannot rely solely on parameter counts. Because repeatedly executing layers amplifies computational overhead, comparisons are only meaningful when conducted under the same pretraining budget and actual training costs.

2026-07-20 ~ 2026-07-21 · 7 related posts

• [source] Loopie Recurrent Transformer Outperforms Traditional Architectures at Equal Compute — Zitian Gao · 2026-07-20
• [source] Loopie: A Looped Transformer with Fewer Parameters — teortaxesTex · 2026-07-20
• Looped Transformer Research and Compute-Matched Scaling — teortaxesTex · 2026-07-20
• Universal Transformer validation cost billions, but MoE scaling still favors layer-loop — teortaxesTex · 2026-07-21
• Loopie uses recurrent MoE layers to beat a compute-matched 30B baseline — burny_tech · 2026-07-21
• A $100M compute study says layer-loop beats model-loop for large MoE pretraining — teortaxesTex · 2026-07-21
• [source] 20B Looping paper says it matches Qwen3 Coder 30B with 10% of pretraining tokens — Dany0 · 2026-07-21