Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x
stanfordnlp · x · 2026-10-21
A companion write-up to Stanford CS336 (Spring 2026) Lecture 6 by Percy Liang breaks down GPU kernel optimization:
- A shorter tensor expression can still mean many memory trips: a constructed 1024×1024 float32 softmax across five passes moves 32 MiB logically, while an ideal fused row kernel moves 8 MiB — a 4x traffic ratio, not a measured speedup (caches, register pressure and spills still matter).
- Triton's model: one program owns a block, masks the tail, computes locally; BLOCKSIZE counts logical elements, not CUDA threads.
- Benchmarking guidance: launches are async, so warm up and time with CUDA events rather than host timers.
- Examples progress from fused GELU to softmax, row reductions, and tiled matmul with activation; official recording and code are public.
Related event: Stanford CS336 Lecture: GPU Kernel Optimization Centers on Memory(2 posts)→
More from Infra
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21
- DeepSeek open-sources DeepInfer, the vLLM-based engine behind V4.1-Flash — The Decoder · 2026-10-21
- Robotics startup launches Deployments to serve fine-tuned HF checkpoints to robots in real time — arnie_hacker · 2026-10-21
- ByteDance, Alibaba, Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B, Reuters reports — SumitGup · 2026-10-21
- TorchTPU demo: unified vLLM & SGLang backend for LLM serving on TPUs — Rob_Mulla · 2026-10-21