Qwen Training Details: Full-Network Muon Optimizer and Residual Stream Design

On August 27, @nrehiew published a thread systematically detailing Qwen's training setup and ablation findings, covering four areas: optimizer, hyperparameters, the Ngram module, and residual stream architecture.

Confirmed

• Optimizer: Muon is used across the network; AdamW is applied to the Router and GR projections for stability, and Adam without weight decay to the Ngram table; per-head orthogonalization is implemented; training uses tensor parallelism (TP).
• Hyperparameter experiments show that changes to the architecture and optimizer shift the best hyperparameters toward larger batch sizes and learning rates. With Muon, no batch size warmup is needed since good gradient noise is achievable with small batches; learning rate tolerance is wider, performing well within a factor of sqrt(2) of the optimum; and learning rate decay slows as model size grows. Stability tests show Muon alone at high learning rates causes extremely high gradient norms and residual activations.
• The Ngram module draws on DeepSeek Engram; ablating its placement across layers found no clear winner, and it was ultimately placed at layer 2 so its computation can be prefetched from CPU during layer 1. The experiments also show that Loss is not a perfect signal: as vocabulary grows, downstream loss trade-offs emerge (the materials do not give the final decision details).
• On residual stream design, widening the residual stream yields substantial gains, with one branch processing input from previous layers and the other three branches local. The documentation compares HyperConnections against simplified variants and introduces their in-house GatedResidual variant.

Why it matters

• Muon used end-to-end in a large-scale production model, paired with targeted stabilization strategies, gives the community first-hand guidance on optimizer choice and tuning—notably the conclusions that no batch size warmup is needed and learning rate tolerance is wider, both of which can substantially cut tuning costs.
• The Ngram module experiments show training Loss doesn't always align with downstream performance, highlighting the risk of selecting designs on Loss alone, with methodological implications for future model design.

2026-08-27 ~ 2026-08-27 · 5 related posts

• [source] Qwen Residual Design: Widening Stream Offers Benefits — nrehiew_ · 2026-08-27
• [source] Qwen Training Details: Muon Usage and TP Load Balancing — nrehiew_ · 2026-08-27
• Ngram Module Experiments: Loss Not a Perfect Signal — nrehiew_ · 2026-08-27
• Muon optimizer eliminates need for batch size warmup — nrehiew_ · 2026-08-27
• [source] Qwen Hyperparam Experiments: Muon Offers Stability — nrehiew_ · 2026-08-27