100% GPU Utilization Can Still Be Slow: A 42-Page Handbook on SMs, Warps and Memory
techNmak · x · 2026-09-13
While writing a 42-page GPU performance handbook, the author realized most people conflate three different things: the work a kernel launches, the warps actually resident on the GPU, and the warps eligible to issue right now.
Key points:
- A kernel with 240 blocks × 256 threads has 61,440 logical threads (1,920 warps), but register/shared-memory/thread limits mean not all are resident at once; resident warps may still wait on memory, dependencies, or sync.
- Occupancy ≠ GPU utilization: occupancy is resident warps vs. architectural max; NVML utilization is just the fraction of the sampling window with at least one kernel executing.
- So a GPU can show 100% utilization while Tensor Cores and HBM bandwidth are far from saturated, with one subsystem bottlenecked.
- The memory hierarchy (registers, shared memory, L1, L2, HBM) differs in scope, capacity, and management—not just speed. A model fitting in HBM says nothing about access efficiency. FlashAttention is a classic example of reorganizing execution scheduling to cut HBM traffic.
- Tensor Cores only do matrix multiply-accumulate; reductions, indexing, elementwise ops, sync, and memory movement still run elsewhere.
The takeaway: stop asking "why isn't the GPU at 100%?" and ask "what is actually limiting useful progress right now?"
Related event: 42-Page GPU Performance Guide Clarifies Utilization Myths(2 posts)→
More from Infra
- Would LITE accepting MRVL's merger offer prove its moat was overstated? — BenBajarin · 2026-10-30
- Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30