NVIDIA explains MoE vs. dense models: how 30B Lightning activates only 3B params per token
NVIDIAAI · x · 2026-09-16
NVIDIA published a technical blog using Nemotron 3.5 Lightning (30B total, 3B active per token) to explain dense vs. MoE architecture trade-offs:
- Decoupled costs: MoE decouples memory from compute — all experts stay in VRAM while only a subset activates per token; dense models scale both together
- Throughput: at equal total params, MoE (Lightning) beats dense (Gemma 4 31B) on token throughput, though the latency edge narrows at high concurrency
- Fine-tuning & quantization: watch for router imbalance; quantization hits router and recurrent-projection layers differently than dense components
- Selection: choose based on memory budget, concurrency, fine-tuning plans and quantization behavior, not raw parameter count
The model is available via build.nvidia.com, Hugging Face, and OpenRouter.
Related event: NVIDIA Explains Dense vs MoE Architecture Trade-offs(2 posts)→
More from Infra
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02