NVIDIA Paper: MoE Interactive Throughput Nearly Doubled
omarsar0 · x · 2026-07-08
omarsar0 highlights an NVIDIA paper on model compression. The work compresses the MoE model Nemotron-3-Super into Puzzle-75B-A9B, roughly doubling interactive serving throughput while preserving quality.
The core innovation is joint structural search: heterogeneous MoE pruning, active parameter budgets, and Mamba pruning are optimized simultaneously rather than sequentially. This is integrated into an iterative pipeline featuring distillation, reinforcement learning, quantization, and multi-token prediction heads. This joint optimization is key to the significant throughput boost under single-GPU interactive latency.
Related event: NVIDIA Open-Sources 75B Parameter MoE Model Puzzle(4 posts)→
More from Infra
- Deep dive: how Weka is breaking Nvidia's AI memory bottleneck — AccBalanced · 2026-10-27
- Anthropic locked into $475B compute deals, OpenAI lifts projected spend to ~$750B through 2030 — kyliebytes · 2026-10-27
- First systematic data center noise study: 17 of 25 hyperscalers drew no credible complaints — AndyMasley · 2026-10-27
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27
- RFD4-Proteina runs on an RTX 3060: protein design from $0.001 per design — AllThingsApx · 2026-10-27
- NPM packages now take ~20 minutes to go live, dev suspects AI-attack throttling — cocktailpeanut · 2026-10-27