Why a 14GB model fitting in an 80GB H100 tells you nothing about inference speed

techNmak · x · 2026-09-14

A worked example shows why "the model fits in VRAM" says little about inference speed: a 7B model at 2 bytes/weight is 14GB, trivially fitting in an H100's 80GB HBM — but streaming all weights at the advertised 3.35TB/s still takes 4.18ms per decode step, capping throughput near 239 tokens/s as a theoretical bound.

Key points:

• Capacity ≠ performance: what matters is which bytes move, where they're served from, how fast, and whether the kernel exposes enough independent work to hide latency. Workloads can fit in HBM yet be bandwidth-bound, or hit high bandwidth yet stall on dependent loads.
• The hierarchy is messier than the pyramid diagram: CUDA memory-space names aren't physical locations — global loads can be served by L1/L2 without touching HBM, and "local memory" actually lives in device memory.
• Reuse is everything: fetching a value once from HBM and reusing it dozens of times from shared memory/registers vs. re-fetching changes performance drastically; high-performance GEMM is as much a data-movement schedule as a matmul algorithm.
• Access geometry matters: the same 256 useful bytes loaded by a warp produce 256/512/1024 bytes of request footprint at stride 1/2/4.
• FlashAttention is the cleanest AI example: the attention function is unchanged — only where intermediate tiles live and whether they make unnecessary HBM trips.

The author compiled these notes into a handbook on the GPU memory hierarchy covering registers, shared memory, L1/L2, HBM, locality, coalescing, tiling, GEMM, attention, and prefill vs. decode.

Original post →

More from Infra

• SK hynix completes HBM4 internal qualification, ushering in custom base die competition — blaizedsouza · 2026-09-14
• One architectural change cuts KV cache 8x: how GQA works, explained with Llama 3 70B — blaizedsouza · 2026-09-14
• A complete breakdown of HBM system architecture, from DDR roots to GDDR7, PIM and HBF alternatives — blaizedsouza · 2026-09-14
• Nvidia paper shows transformer LLMs can be sparser, faster, and lighter without losing accuracy — YesThisIsLion · 2026-09-14
• Team open-sources Tahuna infra to power Hillclimb, an autonomous ML experimentation loop — Monaim101 · 2026-09-14
• Musk: 10 million tons to orbit per year needed for terawatt of space compute — elonmusk · 2026-09-14

Infra channel →