Qwen3.8-Next paper: matches 397B predecessor with 1/9 the training FLOPs

NielsRogge · x · 2026-08-28

NielsRogge (Hugging Face) announced Papers with Code now supports external non-arXiv PDFs: a CRON job queues submissions, converts them to Markdown with the Chandra OCR model, and enables the "chat with paper" feature, running on Hugging Face Jobs. He shared the Qwen3.8-Next architecture paper as an example.

Key points of the paper (a sparse MoE with 125B params, 6B activated per token, plus 51B of n-gram embedding tables off-accelerator):

• With 1/3 the activated params, 1/3 the training tokens and 1/9 the training FLOPs, it beats the 397B-A17B predecessor on 8 of 14 pre-training benchmarks and trails by at most 2.6 points elsewhere
• Token mixing is a layer-wise hybrid of Gated DeltaNet and global attention (one full-attention layer per four); at continued pretraining these are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity via a compressed lightweight indexer
• The residual stream is widened to four branches read through an elementwise gate (Gated Residual)
• Every change is evaluated on loss + downstream benchmarks, training/prefill/decode cost, and effects on optimal hyperparameters and stability; loss and downstream accuracy don't always move together — enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates
• The architecture plus the Muon optimizer shift optimal LR and batch size upward, remove the need for batch-size warmup, and greatly improve stability under stress tests

Related event: Papers with Code Now Supports Non-arXiv Papers with OCR and Chat(2 posts)→

Original post →

More from Models

• Unredacted Anthropic filing exposes internal data-quality debate months before Opus 3 cutoff — zacharynado · 2026-10-15
• Google says teams across the company are using Gemini 4 Argon on its hardest engineering problems — CurieuxExplorer · 2026-10-15
• Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
• 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
• Leak: Moonshot's Kimi flagship 'x4' done with pre-training, K3.1 in internal testing — lyraxana · 2026-10-15
• Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15

Models channel →