FlashAttention kernel numerics are fragile: large-number subtraction problem persists
YouJiacheng · x · 2026-10-04
A technical discussion on attention kernel numerical stability: the author argues the FA kernel implementation is fragile, since exp(sij - lse) is effectively a delta of large numbers, and even the alternative that separates the lse and uses c (S - m) - log(l) still contains a large-number subtraction in (S - m).
Related event: FlashAttention's Numerical Fragility Sparks Discussion(2 posts)→
More from Infra
- Strata on a 4090 48GB: conversation parking buys 35x, n-gram table in RAM just 1% — Shot-Ad-4147 · 2026-10-04
- Cloudflare Workers tip: loop cron work back near your DB, p90 queries drop 75% — DanielLockyer · 2026-10-04
- Aletheia: AMD's $40B server CPU 2027 target supply-constrained, 70% growth seen in 2028 — zephyr_z9 · 2026-10-04
- Caddy: automatic-HTTPS Go web server holds steady with 76k+ stars — caddyserver · 2026-10-04
- Local TTS + BLE remote: voice-controlling Claude from the couch with 500ms latency — Cyborg-2077 · 2026-10-04
- HPE Posts Record $12.2B Revenue, Up 34% YoY, as AI FOMO Drives Hardware Boom — DavidLinthicum · 2026-10-04