SMELT paper: looping middle layers twice in MoE saves 6.8-18% training FLOPs
iScienceLuvr · x · 2026-09-02
New paper SMELT (Sparse MoE Transformer, middle layers Loop Twice) studies looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.
The recipe loops the middle half of layers twice, matching an unlooped baseline on all three budgets. The authors scale across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law per architecture.
Result: SMELT's loss drops faster with compute, saving 6.8–18.0% of training FLOPs on the compute-optimal frontier.
Related event: ByteDance's SMELT: Looping MoE Layers Cuts Training Compute Up to 18%(4 posts)→
More from Research
- Offline Rubric Synthesis Plus Refinement Loops: A Practical Reward Hacking Mitigation — stochasticchasm · 2026-09-22
- Frontend design framed as visual agent task with groupwise relative grading — stochasticchasm · 2026-09-22
- Team reportedly plans to open source 7,000 RL training environments — airesearch12 · 2026-09-22
- Why RL generalizes to reasoning but not literary writing, per AI researchers — phl43 · 2026-09-22
- Rethinking Policy Gradients: Score Centering Skips Importance Sampling Entirely — brandondamos · 2026-09-22
- Building one of the hardest on-policy lie datasets for Aletheia's Quest lie detection competition — hunarbatra · 2026-09-22