42-page GPU performance handbook untangles utilization, occupancy and real bottlenecks

techNmak · x · 2026-09-13

Writing a GPU performance handbook forced the author to separate three things most people conflate: work a kernel launches, work actually resident on the GPU, and work eligible to issue right now.

A 240-block × 256-thread kernel yields 61,440 logical threads (1,920 warps), but register/shared-memory/thread/warp/block limits mean far fewer are resident; the scheduler can only pick from eligible warps.
Occupancy measures resident warps vs. the architectural max, while NVML's "GPU utilization" is just the fraction of the sampling window with at least one kernel running — so a GPU can show 100% utilization while Tensor Cores and HBM bandwidth sit idle.
The memory hierarchy isn't just progressively slower boxes; registers, shared memory, L1, L2 and HBM differ in scope, capacity, management and access behavior. A model fitting in HBM says nothing about bytes moved, coalescing or reuse — FlashAttention is the classic example of reorganizing the schedule to cut HBM traffic.
Tensor Cores handle matrix multiply-accumulate, but reductions, indexing, elementwise ops, synchronization and memory movement still run elsewhere.

The upshot: stop asking "why isn't the GPU at 100%?" and ask "what is actually limiting useful progress right now?" — expanded into a 42-page handbook covering SMs, warps, schedulers, latency hiding, coalescing, GEMM mapping, precision and commonly misread performance numbers.

Original post →

42-page GPU performance handbook untangles utilization, occupancy and real bottlenecks

More from Infra