Looped Transformers work best from scratch, with two passes emerging as the sweet spot
jm_alexia · x · 2026-07-27
Key takeaways from a loop-architecture tech report
The quoted summary highlights several ablation findings about looped Transformers:
- Training from scratch works better than upcycling a standard pretrained Transformer into a looped architecture.
- Two passes seem to be the sweet spot: extra passes bring only marginal gains, while increasing training cost and making optimization less stable.
- KV sharing is not free: it cuts KV-cache memory in half, but performs worse than keeping separate KV states for each pass.
- Depth and width must be co-designed with loop count to balance capability, training cost, and inference efficiency.
The context mentions Nanbeige4.2-3B, a looped Transformer released as a capable 3B agent, and says Nanbeige4.5 is already training with LoopSplit, mHC+depth attention, and concatenated n-gram embeddings.
More from Models
- Claude Opus 5 reportedly nails a snowboarder test in one shot — rohanpaul_ai · 2026-07-27
- GPT 5.6 Pro reportedly beats Work/Codex High and Extra High — latticecut · 2026-07-27
- A developer says Codex is now their coding environment inside ChatGPT — kevinkern · 2026-07-27
- Nebius readies for tomorrow’s Kimi launch, with a bigger team and hiring stakes — demian_ai · 2026-07-27
- Claude Opus 5 ranks third on VoxelBench, just 30 Elo behind the leader — legit_api · 2026-07-27
- Open source and open weights are not the same thing, a post argues — RisingSayak · 2026-07-27