Explained: N-gram vs Experts in Qwen's Architecture
Beamsters · reddit · 2026-08-27
A deep dive into the mechanism combining Mixture of Experts (MoE) and N-gram tables in the Qwen4Exp architecture.
Core Concepts:
- MoE (Experts): Handles arithmetic and reasoning. The routing decision arrives late in the layer, requiring heavy data movement, making it unsuitable for disk storage.
- N-gram Table: Handles memory and recall. Addressed via a hash of the last few tokens, it allows immediate fetching of small data chunks (a few kB), making it suitable for SSD storage.
Metrics & Benefits:
- Qwen 3.8 Flash Next keeps 125B parameters in the MoE network and another 51B in the N-gram table.
- Only 6B parameters are activated per token, retaining the capacity of a 176B model while maintaining the speed of a much smaller one.
- Under a fixed budget, allocating roughly 20-25% of total parameters to the N-gram table yields the most benefit, allowing early layers to handle patterns and deeper layers to focus on reasoning.
More from Infra
- AWS and NVIDIA to Deploy 2 Million More GPUs Through 2027-2028 — rohanpaul_ai · 2026-08-27
- AWS expands NVIDIA roadmap by 2M GPUs for 2027-2028 deployment — rohanpaul_ai · 2026-08-27
- Analyze Nvidia Strategy: Buybacks vs. Heavy HBM Investment — toptickcrypto · 2026-08-27
- TokenSpeed adds Day-0 support for Qwen 3.8 Flash Next architecture — Alibaba_Qwen · 2026-08-27
- Z.ai serving 100T tokens/day on Chinese hardware implies training capability — SumitGup · 2026-08-27
- DLSS 4.5 Ray Reconstruction released with 2nd-gen joint denoiser — ctnzr · 2026-08-27