Meta extends FlashAttention-4 with MXFP8 on Blackwell, hitting 2.85 PFLOP/s forward
PyTorch · x · 2026-09-17
Meta Engineering extended FlashAttention-4 with MXFP8 support for NVIDIA Blackwell—forward and backward kernels, fused quantization, and jagged cross-attention—plus an end-to-end jagged module with fused quantization and FP8 activation/compute, already used internally for GEM training. On latest-gen hardware, the LP FA4 kernel reaches 2.85 PFLOP/s forward and 2 PFLOP/s backward, with up to 1.30× end-to-end module speedup. Design and open-source implementation detailed in their blog.
More from Infra
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02
- Self-Hosting LLMs at a Law Firm: Infra and Data Governance Harder Than the Model — EducationalGroup7245 · 2026-11-02