100% GPU Utilization Can Still Be Slow: A 42-Page Handbook on SMs, Warps and Memory
techNmak · x · 2026-09-13
While writing a 42-page GPU performance handbook, the author realized most people conflate three different things: the work a kernel launches, the warps actually resident on the GPU, and the warps eligible to issue right now.
Key points:
- A kernel with 240 blocks × 256 threads has 61,440 logical threads (1,920 warps), but register/shared-memory/thread limits mean not all are resident at once; resident warps may still wait on memory, dependencies, or sync.
- Occupancy ≠ GPU utilization: occupancy is resident warps vs. architectural max; NVML utilization is just the fraction of the sampling window with at least one kernel executing.
- So a GPU can show 100% utilization while Tensor Cores and HBM bandwidth are far from saturated, with one subsystem bottlenecked.
- The memory hierarchy (registers, shared memory, L1, L2, HBM) differs in scope, capacity, and management—not just speed. A model fitting in HBM says nothing about access efficiency. FlashAttention is a classic example of reorganizing execution scheduling to cut HBM traffic.
- Tensor Cores only do matrix multiply-accumulate; reductions, indexing, elementwise ops, sync, and memory movement still run elsewhere.
The takeaway: stop asking "why isn't the GPU at 100%?" and ask "what is actually limiting useful progress right now?"
Related event: 42-Page GPU Performance Guide Clarifies Utilization Myths(2 posts)→
More from Infra
- Fiber-to-Chip Coupling Needs Path Analysis, Not Single Metrics Like 1dB Loss — jwt0625 · 2026-09-13
- Pentagon reportedly weighs $5B loan to Fluidstack for AI infrastructure supply chain — VraserX · 2026-09-13
- vLLM-Omni makes MiniMax H3 real-time: 10s MP4 in 8.7s on 8x B300, 49 DiT passes cut to 4 — 机器之心 · 2026-09-13
- AMD overtakes Qualcomm to become world's third-largest fabless chip company — xiaosun86 · 2026-09-13
- vLLM Dev Pushes Back as 5 Specialized Inference Engines Launch in One Month — AccBalanced · 2026-09-13
- Ex-compute veteran rebuts Dario's 'pace the frontier': compute doesn't idle, it reroutes — basedjensen · 2026-09-13