From Scratch: Building B200 Attention Kernel to 94.4% of FA4 with 60 Diagrams
JosephJacks_ · x · 2026-08-29
A deep dive into building B200 attention kernels from scratch using CUDA and PTX.
- Goal: Optimize from a naive baseline to achieve 94.4% of FlashAttention-4 (FA4) performance.
- Content: Features 14 kernel iterations and 60 diagrams to visualize each optimization step.
- Progression: Introduces one major optimization at a time (e.g., memory coalescing, Shared Memory, Tiling), pairing concepts with code.
- Application: The final optimized kernel is integrated and tested within a video-generation model.
More from Infra
- First vLLM Conference wraps with a capacity rooftop happy hour co-hosted by AMD — vllm_project · 2026-08-29
- AtomicChat's Qwen3.8-Flash-Next Quant Cuts RAM from 106GB to 65GB, Prefill at 500 t/s — tolitius · 2026-08-29
- Cerebras founder: AI is accelerating hardware evolution and reshaping the industry — Sethwinterroth · 2026-08-29
- Debating between Apple M5 Ultra and RTX 6000 Pro for image/video model inference — Bulky_Astronomer7264 · 2026-08-29
- MCP usage explodes as Agent Handler calls surge 1220x this year — shensi · 2026-08-29
- MiniMax optimizes prompt expansion latency to under 1.5s — isidentical · 2026-08-29