LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou, Zhenzhong Lan, Yankai Lin, Wayne Xin Zhao, Jianguo Li, Chongxuan Li, Ji-Rong Wen
cs.AI
2026-08-04
First scaling laws for MoE diffusion LMs, then LLaDA MoE v2 (30B-A3B, 23.5T tokens): it nears Qwen3 with ~65% of its tokens and beats SDAR Chat on 7 of 8 tasks after SFT.
Diffusion language models (dLLMs) are the main alternative to autoregressive (AR) models like GPT. Instead of writing one token after another, a dLLM masks part of a sequence and reconstructs the masked tokens in parallel, which promises faster decoding and editable generation. The problem is that almost everything known about scaling, batch size, and learning rate comes from AR models, and dLLMs have mostly copied those recipes. The two are not the same optimization problem: a dLLM computes loss only on masked positions, and every prediction conditions on a corrupted sequence rather than a clean causal prefix. Borrowing AR hyperparameters means budgeting in the wrong coordinate system.
The MoE version is even less charted. How many experts to activate, how fine-grained they should be, and how much to put in shared experts are settled questions for AR MoE but nearly untested for dLLMs. This paper runs those tests and uses the answers to train a competitive model.
The authors sweep three scaling dimensions across models from 158M to 3.6B parameters and budgets from 10^18 to 3×10^20 FLOPs.
The first is optimization. The fitted laws are B = 0.374·C^0.3481 for batch size and η = 64.8·C^-0.2447 for learning rate. Compared with DeepSeek's AR law, the dLLM optimal batch grows faster and the learning rate decays faster. At 10^20 FLOPs the dLLM law calls for 3.43M tokens per batch versus 1.02M for AR, and a learning rate of 8.27×10^-4 versus 9.86×10^-4. This lines up with the mechanism: reconstructing many masked positions from noisy input gives noisier gradients, so a larger batch wants a smaller step.
The second is how to split compute between model size and data, via IsoFLOP analysis. The frontiers are M(C) = 0.5152·C^0.475 for model and D(C) = 1.9411·C^0.525 for data, a slight data-side tilt that runs opposite to dense dLLM frontiers, which favor the model side.
The third is MoE structure. At fixed activated compute, lower activation ratio (sparser) wins more as scale grows; expert granularity from 8 to 16 is robust and flat; and a 33.3% shared-expert fraction stays optimal, contrary to AR MoE conventions. The final design is (A, G, S) = (9.09%, 8, 33.3%): 128 routed experts with top-8 routing plus one shared expert at 4× width.
LLaDA MoE v2 is trained from scratch under this recipe: 30B total parameters, 3B activated (30B-A3B), across five stages totaling 23.5T tokens including a long-context anneal to 32K.
Selected pretraining results, against Qwen3 and SDAR Sci:
| Benchmark | LLaDA MoE v2 | Qwen3 | SDAR Sci |
| MMLU | 78.01 | 81.38 | 82.72 |
| MMLU-Pro | 57.28 | 61.49 | 56.96 |
| GSM8K | 83.93 | 91.81 | 86.13 |
| MATH | 54.72 | 59.04 | 48.52 |
| MBPP | 71.00 | 74.40 | 60.40 |
| LiveCodeBench v6 | 31.86 | 49.18 | 39.87 |
It already beats the fellow diffusion model SDAR Sci on MMLU-Pro, MATH, and MBPP, but trails Qwen3 throughout, by nearly 17 points on LiveCodeBench v6. Chinese benchmarks gap wider: CEval 76.11 against Qwen3's 87.50, CMMLU 77.99 against 86.35.
After SFT only (3 epochs, 7M instruction examples, no RL), it beats SDAR Chat on 7 of 8 reasoning and coding tasks. AIME24 is 30.00 versus 16.70, and MultiPL-E at 67.52 edges past Qwen3's 66.60. The gap to Qwen3 narrows but does not close.
This is the first time diffusion language models get their own MoE scaling laws and a structural recipe. Anyone training a dLLM used to copy AR hyperparameters; now batch size, learning rate, the model/data split, and expert configuration all have laws you can apply directly. The practical signal is that a 30B-A3B diffusion model is already touching the Qwen3 tier on a fair share of English knowledge and reasoning, with about 65% of the pretraining tokens. That is the strongest evidence so far that the diffusion route scales and competes.
Approaching Qwen3 still has to be read with the gap attached. English knowledge, math, and mid-difficulty coding are genuinely close; Chinese and live coding are not. Treating diffusion models as drop-in replacements is still premature.
The authors state that the three scaling dimensions were varied separately, so their interactions are untested. That is a real caveat for MoE, where activation ratio and expert granularity likely couple.
Two more gaps remain. The model uses SFT with no RL, so the reasoning ceiling is not pushed and the Qwen3 comparison is not fully like-for-like. And the Chinese and hard-coding gaps are sizeable, pointing to data or strategy weaknesses the paper does not decompose. The Chinese share within 23.5T tokens and the per-stage data mix are not detailed, which blocks reproduction and attribution.