How GPUs really run deep learning: a primer on memory hierarchy and optimization
goyal__pramod · x · 2026-09-25
Damek Davis published a GPU fundamentals and optimization primer distilling months of learning.
- Three performance regimes: memory-bound, compute-bound, plus a third "Overhead" regime.
- Compute/memory imbalance: an A100 delivers 19.5 TFLOPS but only 1.5 TB/s memory bandwidth — over 50 calculations in the time it reads one 4-byte number.
- Two core strategies: operator fusion and tiling, with deep dives into coalesced loads, bank conflicts, warps, occupancy & latency hiding, thread divergence, and quantization.
A distilled index of classics like Making Deep Learning Go Brrrr From First Principles and the CUDA matmul optimization worklog — a solid systematic entry point for GPU performance work.
More from Infra
- Powering a home GPU cluster: one PDU safely feeds ~16 RTX 6000 Max-Q cards — TheZachMueller · 2026-09-25
- Docker launches Cloud Sandboxes: microVM isolation for always-on agents, $250 credit — juntao · 2026-09-25
- Muse Spark 1.3 Now Available via Oracle, in Private Preview on Google Cloud — alexandr_wang · 2026-09-25
- Andy Matuschak: Modern Chips Are Too Durable for Compute-Control-Based AI Governance to Hold — andy_matuschak · 2026-09-25
- Qualcomm touts double-digit MLPerf performance and efficiency gains at Snapdragon Summit — samcharrington · 2026-09-25
- HEIF Heist: image parser RCE chain nets $100k Meta bounty, hits OpenAI repos and more — evilsocket · 2026-09-25