PyTorch's TLX-based JFA kernel beats FlashAttention-4 by 13% fwd, 50% bwd on B200
PyTorch · x · 2026-10-02
PyTorch published a blog detailing Jagged Flash Attention (JFA), the attention kernel behind Meta's Generative Ads Model (GEM), running on NVIDIA Blackwell B200.
- Built with TLX (Triton Low-level Extensions), which add explicit, hardware-aware control on top of Triton's tile-based programming model.
- 3.2K lines of concise Triton-level code — roughly 3× less than the 10K-line CuteDSL kernels of FlashAttention-4 (FA4).
- Outperforms FA4 on the jagged shapes that matter for GEM: 13% faster on forward pass and 50% faster on backward pass.
The post is a strong case for writing near-hand-tuned kernels in a high-level language, with direct relevance for inference/training kernel engineers.
Related event: Meta's 3.2K-line Triton JFA Kernel Beats FA4 on Blackwell(2 posts)→
More from Infra
- Investor: market hates uncertainty more than money after Nvidia earnings — MartinGTobias · 2026-11-19
- SGLang ships Day-0 serving support for new model DaisyRose-27B — ying11231 · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- DaisyRose-27B's in-house sandbox: ~80% cheaper than cloud providers, 99.95% success rate — BanghuaZ · 2026-11-19
- Tencent's 220B Hy4-flash quantized to 49 GB weights, runnable on 64 GB VRAM — gladkos · 2026-11-19
- Berth: a lightweight open-source GPU resource manager with multi-GPU scheduling and VRAM parking — Distinct-Pie2389 · 2026-11-19