FeatherAttn: 4x faster attention backward kernel on AMD Strix Halo

woct0rdho · reddit · 2026-10-31

Developer woct0rdho open-sourced FeatherAttn, an attention backward kernel optimized for AMD RDNA3/3.5 GPUs such as Strix Halo.

Key points:

FlashAttn on RDNA3 routes to Triton kernels in AMD's AITER collection; the forward kernel is fast, but nobody had optimized the backward pass
FeatherAttn's redesigned pipeline is 4x faster at head dim 128 and 1.3x at head dim 64 on Strix Halo
Most recent models (Anima, Krea 2, MOSS, H3) use head dim 128 (SDXL uses 64), so the gains cover current workloads
Tested to make Anima LoRA training 40% faster
No int8/fp8 quantization or sparsity used; exploiting those on RDNA3/3.5 remains an open question

More from Infra