Modular's LLM Inference Handbook adds a GPU architecture primer for kernel work
blaizedsouza · x · 2026-09-22
Modular added a GPU architecture section to its LLM Inference Handbook, aimed at anyone writing or tuning CUDA kernels for LLM inference.
Key argument: without a working mental model of how a GPU runs code, advice like "increase occupancy" or "avoid shared memory bank conflicts" is just rote memorization — you never learn when it applies.
What it covers:
- Hardware: streaming multiprocessors, Tensor Cores, and the memory hierarchy of registers, caches, shared memory, and HBM
- Execution model: threads organized into warps (32 threads), blocks, and grids, and how the GPU schedules them
- Kernel optimization: mapping the execution model onto hardware under limits on compute, bandwidth, registers, shared memory, and scheduling
Details lean toward NVIDIA hardware given CUDA's dominance, but concepts apply to AMD GPUs and other accelerators. All pages are readable as Markdown by appending .md to URLs, with an llms.txt index.
More from coding & agent
- Engram: A Local Encrypted Memory Vault Unifying Agent Memory Across AI Tools — Acceptable_Leg3950 · 2026-09-22
- Sol launches an email agent that finds every promise in your inbox and starts the work for you — alexmacgregor__ · 2026-09-22
- Yacine: your Linux dotfiles could just be words, with apps built by AI on demand — yacineMTB · 2026-09-22
- Token Saver: Claude Skills Route Coding Work From Expensive Codex to Cheap Muse 1.3 — AIandDesign · 2026-09-22
- dotey's 4 rules for managing docs in AI coding projects — dotey · 2026-09-22
- Dev: Opus 5 is 'dumb as bricks' for coding, I'm switching to ChatGPT — rickasaurus · 2026-09-22