On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu
cs.CL
2026-08-31
A 125B sparse MoE with 6B activated plus 51B host n-gram tables. Leads a 397B-A17B predecessor on 8 of 14 pretraining benchmarks at about one-ninth the training FLOPs.
The previous Qwen flagship was 397B total with 17B activated per token. The next generation needs to keep that quality while cutting activated parameters, training tokens and FLOPs together. Any architectural change hits three things at once: downstream scores, training/prefill/decode cost, and the optimal hyperparameters plus stability. This report treats those as one design problem and scores every candidate on all three axes.
Loss and downstream accuracy do not always move together. A larger n-gram vocabulary lowers loss monotonically while accuracy saturates. Making residual reads and writes data-dependent barely moves loss and clearly lifts benchmarks. Watching loss alone would have picked the wrong design.
Token mixing is a layer-wise hybrid: three Gated DeltaNet layers and one full-attention layer in every four. GDN compresses the prefix into a fixed-size recurrent state and applies a decay gate plus a delta write, so the decode cache no longer grows linearly. The periodic full-attention layer keeps direct token-level retrieval that no finite-state memory quite reproduces. During continued pretraining those full-attention layers become Qwen Sparse Attention: a lightweight indexer scores micro-blocks and expands them into a sparse core mask, cutting indexer cost from O(n²) to O(n²/r). The shipped setup uses a 2048-token budget and compression ratio 4. QSA first distills the dense attention distribution, then jointly trains the backbone under sparsity; the LM-loss gap on 256K context is on the order of 10⁻⁴.
The residual stream is widened to four branches and read through an elementwise sigmoid gate, called Gated Residual. Width adds capacity; the gate decides how that capacity is spent and supplies the rescaling that keeps training stable. The branch-mixing matrix is dropped, which saves a full residual read per block and helps decode bandwidth.
Extra capacity sits outside the backbone: a single n-gram embedding layer at layer 2, keyed by short n-grams, stored in host memory and prefetched asynchronously. One layer beats multi-layer splits. Growing the vocabulary lowers loss monotonically and helps Chinese benchmarks; other downstream scores saturate.
Muon orthogonalizes two-dimensional linear maps. Embeddings, the output head, the MoE router and GR's low-rank projections stay on AdamW. Fused parameters are split before orthogonalization. The new architecture shifts the optimal learning rate and batch size upward. Batch-size warmup is no longer useful and costs 18.8% extra optimizer steps.
The base model is compared with Qwen3.7-Plus-Base (397B/17B) and Qwen3.8-27B-Base:
| Benchmark | Flash-Next 6B act. | 27B dense | Plus 17B act. |
| MMLU | 90.36 | 87.51 | 90.43 |
| MMLU-Pro | 73.23 | 68.60 | 70.90 |
| SuperGPQA | 51.36 | 44.86 | 48.42 |
| MATH | 72.78 | 60.54 | 74.38 |
| MultiPL-E | 79.09 | 74.50 | 81.68 |
| MGSM | 89.33 | 86.37 | 85.42 |
It leads Plus on 8 of 14 tasks and trails the rest by at most 2.59 points (MultiPL-E). At 25B-A3B, the GDN hybrid averages 53.81 against 49.87 for full attention and 51.15 for an SWA hybrid. At 1M context QSA is 7.6× faster in prefill and 4.9× faster in decode at the attention module; RULER beyond 512K moves from 90.08 to 93.00, MRCR at 512K from 30.66 to 40.53. Under a 4×-optimal learning-rate stress test the old structure spikes often and the new recipe stays flat. The production run had no loss spike and did not use qk-clip or SwiGLU-clip.
This is a design memo that binds architecture, kernels, optimizer and stability, not a single-module paper. Long-context serving teams can copy three pieces: one retrieval layer every four, QSA's micro-block indexer, and n-gram tables in host memory as almost-free parameters. Muon plus gated residuals shift the optimal learning rate and batch size up, so the old warmup habit can go.
For people who only tune LoRA, the transferable lesson is that loss and downstream scores can disagree, and that stability itself changes the optimal hyperparameters.
The base model trails Plus by 1.60 on MATH and 2.59 on MultiPL-E; coding and contest math are not fully covered by the smaller activation. Under a fixed parameter budget, replacing MoE experts with n-gram slots barely helps downstream scores, so the tables behave like bolted-on memory rather than a backbone substitute. QSA numbers come from continued pretraining; from-scratch pretraining is not reported. The stress test recreates instability at moderate scale with a high learning rate, which is still an extrapolation to the full run. Post-training alignment and chat quality are mostly out of scope.