Loopie Cyclic Transformer Beats 30B Baseline with Equal Compute

Zitian Gao's team released Loopie, a recurrent Transformer series that reuses network layers to improve both parameter and compute efficiency. The series includes two MoE versions: 20B total params (2B active) and 6B total (0.6B active). Claimed to outperform a 30B baseline under the same training wall-clock time and matched inference compute budget.

Key Details and Breakthrough

Previously, recurrent Transformers underperformed compared to scaling parameters when increasing pretraining compute. First author Zitian Gao stated the team invested hundreds of millions of dollars in compute to validate the Universal Transformer approach (model-loop), finding it hard to scale to billion-parameter MoE architectures. In late pretraining, their proposed Layer-loop scheme (Loopie) showed superior scalability, overcoming previous pain points.

Evaluation Perspective

@teortaxesTex cautioned that evaluating recurrent Transformers should not focus on parameter count alone. Because layer reuse amplifies compute cost, fair comparison must use identical pretraining budget and actual training cost.

2026-07-20 ~ 2026-07-21 · 6 related posts

• [source] Loopie Recurrent Transformer Outperforms Traditional Architectures at Equal Compute — Zitian Gao · 2026-07-20
• [source] Loopie: A Looped Transformer with Fewer Parameters — teortaxesTex · 2026-07-20
• Looped Transformer Research and Compute-Matched Scaling — teortaxesTex · 2026-07-20
• [source] Universal Transformer validation cost billions, but MoE scaling still favors layer-loop — teortaxesTex · 2026-07-21
• Loopie uses recurrent MoE layers to beat a compute-matched 30B baseline — burny_tech · 2026-07-21
• A $100M compute study says layer-loop beats model-loop for large MoE pretraining — teortaxesTex · 2026-07-21