SMELT: Tsinghua × ByteDance show looped MoE Transformers save 6.8–18% training FLOPs

teortaxesTex · x · 2026-11-21

SMELT (arXiv:2609.01343, Tsinghua × ByteDance Seed) studies looped Transformers under strict compute matching: the middle half of MoE layers is looped twice while matching per-token FLOPs, params, and KV cache.

A practical recipe showing depth reuse yields measurable gains even under budget matching.

Original post →

More from Infra

Infra channel →