Looped Transformer Debate Revives Years-Old Sparse Universal Transformer Work
Amid renewed discussion of looped transformers and per-token adaptive computation, Yikang Shen points out that his earlier Sparse Universal Transformer paper already explored this idea, using SMoE to make shared-parameter transformers scalable.
2026-09-04 ~ 2026-09-04 · 2 related posts
- Looped transformer debate revisited: Sparse Universal Transformer (EMNLP 2023) got there first — Yikang_Shen · 2026-09-04
- Sparse Universal Transformer: SMoE makes parameter-sharing transformers scalable (EMNLP 2023) — Yikang_Shen · 2026-09-04