GB300 NVL72 runs DeepSeek Flash at ~1000 tok/s/user, ~$0.07 per 1M tokens TCO
bookwormengr · x · 2026-10-30
- SemiAnalysis reports that on NVIDIA GB300 NVL72, SGLang achieves close to 1000 tok/s per user on DeepSeek Flash v4.1 by aggressively fusing kernels together.
- @bookwormengr does the math: even at this interactivity level, TCO works out to 13M tokens per dollar, or about 7 cents per 1M tokens.
- The post claims this is achievable with just one NVL72 tray (a 4-GPU-class setup), which matters for defensive cybersecurity, speech, and design applications.
- TileRT started the megakernel engine trend on OSS frontier models, and SGLang is now widely adopting it.
More from Infra
- Rogers Expands Starlink Satellite Roaming to Japan, Australia and New Zealand — XFreeze · 2026-10-30
- Corey Quinn flags macOS update silently restoring the removed eMudhra root cert — kingslyj · 2026-10-30
- NVIDIA shares guide on maximizing memory efficiency and cutting costs on Jetson — NVIDIAAI · 2026-10-30
- Rebuild NVIDIA's speaker-captions demo with one transformers serve command — yb2698 · 2026-10-30
- Cap moves transcription to its own GPU farm, up to 11x faster — JosephJacks_ · 2026-10-30
- Prime Intellect makes DeepSeek's MegaMoE trainable, up to 55% faster MoE training — samsja19 · 2026-10-30