Free 44-session GPU kernel engineering series: naive GEMM to FlashAttention on H100 and Blackwell
Winter_Mistake_3185 · reddit · 2026-10-20
A developer shared a free, code-first GPU kernel engineering series (44 planned sessions) with intro and roadmap videos live and all code public, explicitly not for sale.
Core thesis: for most ML workloads, kernel performance is a data-movement problem before a compute problem. The series builds on two threads:
- Roofline model: attainable throughput is bounded by min(peak compute, memory bandwidth × arithmetic intensity); a kernel's position relative to the ridge point tells you whether to optimize memory access or compute.
- FlashAttention as worked example: mathematically identical attention; the gain comes from tiling so the full N×N score matrix is never written to HBM. Built live from scratch.
Roadmap: CPU parallelism and GPU architecture, memory hierarchy, GEMM from naive to optimized with profiling at every step, tensor cores, profiling/debugging, PyTorch integration and Triton, FlashAttention, and Hopper/Blackwell topics (TMA, WGMMA, CUTLASS, FP8/quantization, NVFP4, multi-GPU basics, FlashAttention 4, DeepSeek's kernels, AI-written kernels, capstone).
Each session follows concept → live code → profile → verified speedup against baseline. Prerequisites: C/C++, basic linear algebra, PyTorch. Later sessions (especially Blackwell/NVFP4) may shift as tooling evolves; the author is soliciting feedback from production kernel engineers.
More from Infra
- Celesto launches cloud computers for agents, claims 96% cheaper CI than GitHub runners — aniketmaurya · 2026-10-20
- NVIDIA's first GB300 FP4 counter-submission still 1.64x worse perf per dollar than TPUv7 — xennygrimmato_ · 2026-10-20
- Long queues at RTX Spark SF launch for local Qwen demos built with ODS software — TheZachMueller · 2026-10-20
- Dream-1 Beta runs robot tasks at $1.72 per success, up to 41x cheaper than LLM agents — gharik · 2026-10-20
- Arm AGI CPU posts 2x end-to-end agentic workflow performance vs leading x86 in Signal65 tests — ryanshrout · 2026-10-20
- Loop engineering is a teeter-totter: you never have enough CPU cores and inference at once — kieranklaassen · 2026-10-20