Large-number subtraction haunts all softmax kernels; maybe fwd-bwd consistency is all we need
YouJiacheng · x · 2026-10-04
Continuing the discussion, the author notes the large-number subtraction problem persists even without FlashAttention, since stable softmax always requires subtracting the max. He suggests maybe we only need forward-backward consistency, expressing some resignation about the fundamental numerical issue.
Related event: FlashAttention's Numerical Fragility Sparks Discussion(2 posts)→
More from Research
- A gentle introduction to phase-locked loops, the Kuramoto model, and gyrotrons — stevenstrogatz · 2026-10-04
- ICLR's official .bib template has listed Bengio twice since 2019 — tughanbulut · 2026-10-04
- kalomaze: In LoRA, Weight Decay Behaves Like Regularizing Toward the Base Model — kalomaze · 2026-10-04
- Humans are vastly more sample-efficient than LLMs, and finding why could rival attention — burny_tech · 2026-10-04
- Tsinghua and Peking University Probe Whether LLMs Think Beyond Language in Nature MI — jiqizhixin · 2026-10-04
- Judea Pearl Announces Second Edition of The Book of Why, Due October 20 — yudapearl · 2026-10-04