Qwen Model Integrates N-gram and Sparse Attention, Validating New Tech with 125B Parameters
rsasaki0109 · x · 2026-08-29
Core Architecture: Qwen3.8-Flash-Next is a 125B parameter MoE model integrating recent promising techniques, activating 6B parameters per token.
Tech Integration:
- N-gram Embedding: Similar to DeepSeek Engram, using 51B params to hash and retrieve N-gram representations, enabling prefetch from slower memory.
- Qwen Sparse Attention (QSA): Selects important context blocks first, then applies detailed attention.
- Gated Residual: Manages multiple residual streams via gating, similar to Hyper-Connections.
Optimization: Uses Muon for matrix parameters and AdamW for others. Re-derives scaling laws based on arch changes, enabling larger LR/batch sizes and achieving high performance.
More from Models
- Opus 5 might be a very deep model or use looping, commentator speculates — scaling01 · 2026-10-16
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- Nvidia-backed post-training pushes Nemotron 3.5 VL past frontier on dermatology diagnosis — LoganGrasby · 2026-10-16
- Gemini 4 Argon scores 97.5% on ARC-AGI-3, second lab above 95% — patience_cave · 2026-10-16
- Microsoft MAI claims SOTA 3T-param in-house base models, post-training results coming — soldni · 2026-10-16
- Datadog's Scout 1.0: post-trained GLM-5.3 Flash matches frontier models for AI SRE at lower cost — Tim_Dettmers · 2026-10-16