NVIDIA GB300 Sets World Record at 1,648 TFLOPs Pre-training DeepSeek-V3
PyTorch · x · 2026-08-07
PyTorch and NVIDIA jointly announced that leveraging the PyTorch native training stack TorchTitan and NVIDIA's optimizations, they set a world record pre-training DeepSeek-V3 671B on the NVIDIA GB300 NVL72 system.
- Performance: Achieved 1,648 TFLOPs per GPU, delivering approximately 6x higher delivered performance.
- Architecture: Enabled by 5th-gen NVLink and high-bandwidth networking, effectively resolving communication bottlenecks in large-scale MoE scaling.
- Scaling Efficiency: Per-GPU throughput held above 97% when scaling from 256 to 1,024 GPUs, demonstrating the power of hardware-software co-design.
More from Infra
- The Truth Behind MiniMax 2x Speedup: Newer Python Dependencies Introduced convrot Support — lmpdev · 2026-08-07
- Musk: Terafab AI Factory Phase 1 to Create Over 3,000 Jobs — elonmusk · 2026-08-07
- Meta Allegedly Building Own Search Engine, Scraping Web at Massive Scale — churchkey · 2026-08-07
- SanDisk Beats Earnings but Stock Falls as AI Inference Drives Memory Demand — ryanshrout · 2026-08-07
- Databricks CEO: Agent Traffic Will Be 1000x Human Traffic in 5 Years — threepointone · 2026-08-07
- Run 2.78T Params on 8GB RAM: Open-Source C99 Engine for Kimi K3 — dr_cintas · 2026-08-07