Qwen3.8-Next paper: matches 397B predecessor with 1/9 the training FLOPs

NielsRogge · x · 2026-08-28

NielsRogge (Hugging Face) announced Papers with Code now supports external non-arXiv PDFs: a CRON job queues submissions, converts them to Markdown with the Chandra OCR model, and enables the "chat with paper" feature, running on Hugging Face Jobs. He shared the Qwen3.8-Next architecture paper as an example.

Key points of the paper (a sparse MoE with 125B params, 6B activated per token, plus 51B of n-gram embedding tables off-accelerator):

With 1/3 the activated params, 1/3 the training tokens and 1/9 the training FLOPs, it beats the 397B-A17B predecessor on 8 of 14 pre-training benchmarks and trails by at most 2.6 points elsewhere
Token mixing is a layer-wise hybrid of Gated DeltaNet and global attention (one full-attention layer per four); at continued pretraining these are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity via a compressed lightweight indexer
The residual stream is widened to four branches read through an elementwise gate (Gated Residual)
Every change is evaluated on loss + downstream benchmarks, training/prefill/decode cost, and effects on optimal hyperparameters and stability; loss and downstream accuracy don't always move together — enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates
The architecture plus the Muon optimizer shift optimal LR and batch size upward, remove the need for batch-size warmup, and greatly improve stability under stress tests

Original post →

Qwen3.8-Next paper: matches 397B predecessor with 1/9 the training FLOPs

More from Models