FlashAttention's Numerical Fragility Sparks Discussion

A developer highlights that FlashAttention's exp(sij - lse) is essentially catastrophic cancellation, and notes the issue persists even without FlashAttention since stable softmax always subtracts the max, suggesting forward-backward consistency as a possible fix.

2026-10-04 ~ 2026-10-04 · 2 related posts

• FlashAttention kernel numerics are fragile: large-number subtraction problem persists — YouJiacheng · 2026-10-04
• Large-number subtraction haunts all softmax kernels; maybe fwd-bwd consistency is all we need — YouJiacheng · 2026-10-04