Are URM and Universal Transformers the forgotten architecture beating standard LLMs?

moschles · reddit · 2026-10-09

Redditor moschles asks whether Universal Transformers (UT) and Universal Reasoning Models (URM) have been quietly absorbed into frontier models, or left as forgotten papers.
UT replaces stacked layers with recurrent computation over depth, repeatedly applying one transition block (H^(t+1) = LayerNorm(H^t + MHA(H^t)) plus a position-wise Transition), with 2D sinusoidal embeddings encoding both position and refinement depth.
The cited paper (arXiv 2512.14693) claims UT-based small models, trained from scratch without internet-scale pretraining, consistently outperform most standard Transformer LLMs by a significant margin. URM adds fixed loops, ACT loops, a ConvSwiGLU module, and a novel Truncated Backpropagation Through Loops scheme.
The post links the paper, a simplified blog, and a YouTube talk, inviting discussion on whether big labs already use these enhancements.

More from Models