Looped Transformer SMELT cuts training FLOPs 6.8-18%, tipped for GPT-6
mark_k · x · 2026-09-03
A paper from Tsinghua, ByteDance Seed and TokenWave tests SMELT, a MoE Looped Transformer that runs the middle half of its layers twice — matched against normal Transformers on FLOPs, params and KV cache.
Key findings:
- 6.8-18% fewer training FLOPs to reach the same loss
- Up to 20.4% compute efficiency gains on coding; benefits grow with longer sequences and more in-context examples
- On the second pass, attention shifts away from attention-sink tokens toward relevant content — the model refines rather than repeats
The author notes OpenAI's imminent Astra / GPT-6 is reportedly based on this architecture (unverified), and looping could be a defining architectural change of next-gen frontier models.
Related event: ByteDance's SMELT: Looping MoE Layers Cuts Training Compute Up to 18%(4 posts)→
More from Models
- Anthropic investigates elevated errors across Claude Mythos 5.1, Fable 5.1 and Opus 5 — ClaudeAI-mod-bot · 2026-09-22
- Dev Swaps Opus for Mimo-v2.6 in Cline on Client Projects: 'It's a Beast' — MicahBerkley · 2026-09-22
- Kev refactored onto Qwen3.5: open-source decision models now at 0.8B, 4B and 9B — alexcovo_eth · 2026-09-22
- Why OpenAI bets on math: it's the most verifiable domain for reinforcement learning — burny_tech · 2026-09-22
- Open-source decision model Laya ported to Core ML: 99.5% ops on ANE, 3.7ms per decision — alexcovo_eth · 2026-09-22
- Fireworks: routing 18 models per task hits 97.6% solve rate at $1.88 vs best single model's 74.1% at $6.52 — sophiamyang · 2026-09-22