Memory-bound at batch 1, compute-bound at batch N: how GPUs actually run LLMs
techNmak · x · 2026-10-23
techNmak published a detailed thread with a full infographic explaining how GPUs execute AI workloads. Key point: the same LLM is memory-bound at batch size 1 and may become compute-bound at larger batches, since low-batch decoding spends more time moving weights and KV-cache than doing math — so FLOP counts can't predict real speed.
Covered topics:
- CUDA execution model: kernel grids, thread blocks on SMs, warps of 32, warp schedulers.
- Memory coalescing: a 32-byte stride makes 32 threads touch 32 sectors (1,024 bytes) instead of 4 (128 bytes) — 8x the DRAM traffic for the same useful data.
- Matmul tiling and Tensor Cores: reuse in registers/shared memory; keeping Tensor Cores busy needs right tile sizes and layouts.
- Kernel fusion and FlashAttention: IO-aware tiling plus online softmax avoid materializing the full attention matrix in HBM, though quadratic complexity remains.
- Occupancy vs throughput, roofline model, and profiling with Nsight Systems/Compute to find launch gaps, warp stalls, and memory transaction inefficiencies.
More from Infra
- Intel CTO confirms partnership with Musk on Terafab: 'We either build it or we don't have the chips' — elonmusk · 2026-10-23
- Researcher: Putting a $200K GPU Server Purchase Order in a Paper Envelope — HildeKuehne · 2026-10-23
- Intel and AMD unite PyTorch CPU optimizations in oneDNN for a common x86 path — PyTorch · 2026-10-23
- Google and Meta open-source TorchTPU, a native PyTorch backend for TPUs — PyTorch · 2026-10-23
- Intel demos AI Playground Home Agent: control local PyTorch compute from your phone via Slack — PyTorch · 2026-10-23
- TorchTPU and TorchNeuron Backends Land as PyTorch Becomes AI's Hardware Battleground — PyTorch · 2026-10-23