Explained: N-gram vs Experts in Qwen's Architecture
Beamsters · reddit · 2026-08-27
A deep dive into the mechanism combining Mixture of Experts (MoE) and N-gram tables in the Qwen4Exp architecture.
Core Concepts:
- MoE (Experts): Handles arithmetic and reasoning. The routing decision arrives late in the layer, requiring heavy data movement, making it unsuitable for disk storage.
- N-gram Table: Handles memory and recall. Addressed via a hash of the last few tokens, it allows immediate fetching of small data chunks (a few kB), making it suitable for SSD storage.
Metrics & Benefits:
- Qwen 3.8 Flash Next keeps 125B parameters in the MoE network and another 51B in the N-gram table.
- Only 6B parameters are activated per token, retaining the capacity of a 176B model while maintaining the speed of a much smaller one.
- Under a fixed budget, allocating roughly 20-25% of total parameters to the N-gram table yields the most benefit, allowing early layers to handle patterns and deeper layers to focus on reasoning.
More from Infra
- ASML CFO says US Match Act restricting DUV sales to China would backfire — pstAsiatech · 2026-10-15
- Bloomberg breakdown: a 1GW Texas AI data center may need nearly $3B for overruns alone — rohanpaul_ai · 2026-10-15
- 7 System Design Algorithms to Know: Merkle Trees to CAP/PACELC — goyalshaliniuk · 2026-10-15
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Sipeed MaixStation product page goes live for edge AI dev — xlltt · 2026-10-15
- Nvidia Is Now Worth More Than the Entire Indian Stock Market — theteknosaur · 2026-10-15