NVIDIA explains MoE vs. dense models: how 30B Lightning activates only 3B params per token
NVIDIAAI · x · 2026-09-16
NVIDIA published a technical blog using Nemotron 3.5 Lightning (30B total, 3B active per token) to explain dense vs. MoE architecture trade-offs:
- Decoupled costs: MoE decouples memory from compute — all experts stay in VRAM while only a subset activates per token; dense models scale both together
- Throughput: at equal total params, MoE (Lightning) beats dense (Gemma 4 31B) on token throughput, though the latency edge narrows at high concurrency
- Fine-tuning & quantization: watch for router imbalance; quantization hits router and recurrent-projection layers differently than dense components
- Selection: choose based on memory budget, concurrency, fine-tuning plans and quantization behavior, not raw parameter count
The model is available via build.nvidia.com, Hugging Face, and OpenRouter.
Related event: NVIDIA Explains Dense vs MoE Architecture Trade-offs(2 posts)→
More from Infra
- SentencePiece Lite ships: 50KB binary, 20-30x faster tokenization for edge devices — heiga_zen · 2026-09-16
- Leaked 10,000-word Huawei memo: become the NVIDIA for any LLM, pivot around Ascend 950 — pstAsiatech · 2026-09-16
- Running Qwen 27B and DeepSeek v4 Flash together on one heterogeneous machine — samsja19 · 2026-09-16
- JPMorgan sees 25M+ GPU/ASIC shipments by 2028, ASICs dominate — a 'narrative violation' — bookwormengr · 2026-09-16
- iamtrask: The Endgame Is a Trust Web of Personal LLM Servers, Not One AGI — iamtrask · 2026-09-16
- Program-as-Weights: 0.6B interpreter matches Qwen3-32B prompting with 1/50 the memory — yuntiandeng · 2026-09-16