SMELT: Tsinghua × ByteDance show looped MoE Transformers save 6.8–18% training FLOPs
teortaxesTex · x · 2026-11-21
SMELT (arXiv:2609.01343, Tsinghua × ByteDance Seed) studies looped Transformers under strict compute matching: the middle half of MoE layers is looped twice while matching per-token FLOPs, params, and KV cache.
- Scaling laws fit at four sizes up to 54B non-embedding params show 6.8–18.0% training-FLOP savings on the compute-optimal frontier, growing with scale
- Downstream gains exceed what validation loss predicts, largest on Code, and grow with sample length and in-context examples
- Mechanistic analysis: the second pass reduces attention sink and redirects mass to content-relevant tokens
A practical recipe showing depth reuse yields measurable gains even under budget matching.
More from Infra
- 2M canary tokens expose stolen prompts and compromised AI providers — AccBalanced · 2026-11-21
- Unsloth release runs 26.4B Inkling 2 Mini on 16GB RAM, cuts MCP token use 87% — danielhanchen · 2026-11-21
- Seroter Daily Reading List #914: harnesses are cheap to swap, model behavior isn't — rseroter · 2026-11-21
- NVFP4 MoE experts force CPU fallback in llama.cpp; requantizing boosts Metal speed 5.8x — teortaxesTex · 2026-11-21
- Compute overbuild skeptic recants: efficiency gains cut AI compute cost ~4x per year — teortaxesTex · 2026-11-21
- AscendGym paper draws attention as Huawei compute platform research picks up — willknight · 2026-11-21