Qwen Details Training Recipes: Muon Optimizer and Residual Stream Findings

On August 27, @nrehiew published a series of posts systematically disclosing Qwen's training details and ablation findings across four areas: optimizer, hyperparameters, the Ngram module, and residual stream design.

Confirmed

• Muon is used across the entire network; the Router and GR projections use AdamW for stability, and the Ngram table uses Adam without weight decay. Per-head orthogonalization is applied, and training uses tensor parallelism (TP).
• Hyperparameter experiments show architectural and optimizer shifts move optimal hyperparameters toward larger batch sizes and learning rates. With Muon, no batch-size warmup is needed since small batches already yield good gradient noise; learning-rate tolerance is wider, performing well within a factor of sqrt(2) of the optimum; LR decay slows as model size grows. Stability tests show that Muon alone at high learning rates leads to very large gradient norms and residual activations.
• The Ngram module, inspired by DeepSeek's Engram, showed no clear winner across placement-layer ablations; it was placed at layer 2 so it can be prefetched from CPU during layer-1 computation. Experiments show loss is not a perfect signal: as vocabulary grows, downstream loss exhibits trade-offs (final resolution not given in the material).
• Widening the residual stream brings substantial gains: one branch processes input from previous layers while the other three are local. The write-up compares HyperConnections with simplified variants and introduces a homegrown GatedResidual variant.

Why it matters

• Muon being used end-to-end in a production-scale model, paired with targeted stabilization tricks, offers first-hand guidance for optimizer choices—especially the conclusions that no batch-size warmup is needed and learning-rate tolerance is wider, both of which can substantially cut tuning costs.
• The Ngram experiments show training loss and downstream performance do not always agree, cautioning against selection based on loss alone—a methodological lesson for future model design.

2026-08-27 ~ 2026-08-27 · 5 related posts

• Qwen Residual Design: Widening Stream Offers Benefits — nrehiew_ · 2026-08-27
• [source] Qwen Training Details: Muon Usage and TP Load Balancing — nrehiew_ · 2026-08-27
• [source] Ngram Module Experiments: Loss Not a Perfect Signal — nrehiew_ · 2026-08-27
• Muon optimizer eliminates need for batch size warmup — nrehiew_ · 2026-08-27
• [source] Qwen Hyperparam Experiments: Muon Offers Stability — nrehiew_ · 2026-08-27