Modal's free GPU Glossary explains how H100 runs 16,000+ truly parallel threads vs CPU's 384
josh_wills · x · 2026-10-25
Modal published the GPU Glossary, a free reference (PDF available, plus a print edition) covering GPU device hardware, the CUDA programming model, host software, and performance concepts.
A striking comparison from the book: an AMD EPYC 9965 CPU draws up to 500W with 192 cores and at most 384 parallel threads (1.25W each), while an H100 SXM GPU draws 700W with 132 SMs, each with four Warp Schedulers issuing to 32-thread warps — over 16,000 truly parallel threads at 5 milliwatts apiece, and 250,000+ concurrent threads across the card.
Key insight: GPU SMs are simpler than CPU cores (no speculative execution), but warp switches happen within a single clock cycle — over 1000x faster than CPU context switches — hiding memory latency and keeping CUDA/Tensor Cores utilized. That latency-hiding, rather than big caches and branch prediction, is the secret to GPU performance.
More from Infra
- Jensen Huang unveils Vera Rubin DSX POD at GTC Berlin, pitches Europe's AI economy — nvidia · 2026-10-25
- TensorSharp ships Day-1 MiniMax-M3.1-Flash support: 9.8× faster 64K prefill than llama.cpp on 8× A40 — fuzhongkai · 2026-10-25
- ANEMLL Forge 0.2.1 runs Qwen3.8-27B at up to 192K context on M6 Mac Mini's Neural Engine — Competitive-Bake4602 · 2026-10-25
- Compute contracts vs reality: Nscale has only 5% of contracted GPUs live, TeraWulf 12% — AccBalanced · 2026-10-25
- Live MCP rewrite from Node to Rust cuts memory usage by roughly 8x — draginol · 2026-10-25
- 305M-Parameter Sparse MoE Runs Computer-Use Agent Fully Offline on a MacBook Air — lmoroney · 2026-10-25