42-page GPU architecture handbook explains why 100% utilization doesn't mean saturation
techNmak · x · 2026-10-18
Engineer techNmak released a 42-page handbook (with a one-page visual overview) untangling common GPU performance misconceptions:
- Separate three layers: launched work, work actually resident on the GPU, and work eligible to issue right now. A kernel with 240 blocks × 256 threads = 61,440 logical threads (1,920 warps), but residency is gated by registers, shared memory, and warp/block limits.
- Metric traps: occupancy is a residency ratio (resident warps per SM vs. max), while NVML's GPU utilization only measures the fraction of a sampling window with at least one kernel running — a GPU can show 100% utilization with Tensor Cores and HBM bandwidth idle.
- Memory hierarchy: registers, shared memory, L1/L2, and HBM differ in scope and management; a model fitting in HBM says little about coalescing or reuse. FlashAttention's tiling + recomputation illustrates reducing HBM traffic.
- Tensor Cores aren't the whole story: reductions, elementwise ops, synchronization, and kernel launch overhead run elsewhere.
The author's takeaway: stop asking "why isn't the GPU at 100%?" and ask "what is actually limiting useful progress right now?" Topics covered include SMs/warps/schedulers, latency hiding, GEMM mapping, and precision.
More from Infra
- Netflix engineer open-sources Headroom, cutting LLM token usage by up to 95% — VeryWellVersed · 2026-10-18
- Book-length deep dive: virtual memory, page tables, TLBs and Linux internals — abhi9u · 2026-10-18
- Microsoft cancels UK data centre deal with Nscale just ahead of its $35B IPO — NinaDSchick · 2026-10-18
- Megabake compiler hits 5.5x torch.compile speedup on SmolLM — ricklamers · 2026-10-18
- Two pallets of ZOTAC RTX 5090 cards worth $470,000 vanish at São Paulo airport — chemist_slime · 2026-10-18
- Muse Spark runs on a single Mac Studio M3 Ultra at 18 t/s decode — challis88ocarina · 2026-10-18