KohakuFA: Blackwell flash attention kernel fixes 10x-1000x gradient bugs in FA4/cuDNN
bdsqlsz · x · 2026-10-04
KBlueleaf's team found that mainstream attention kernels — cuDNN, FlashAttention-4, and flexattention — all suffer severe backward-pass precision issues: once an attention layer learns near-one-hot attention and logits exceed 1e5, gradients are wrong by 10x to 1000x (or inf) while forward and loss look normal. Many "unstable fp16 training" failures were the kernel's fault, not fp16's. They released KohakuFA, a flash attention implementation for NVIDIA Blackwell (sm100+) written in Gluon (Triton's tile-level dialect) using tcgen05 MMAs, TMA, and warp specialization, supporting dense, token-causal, and block-causal attention. Its fp16 gradients stay at the fp16 rounding floor, close to fp64 and FlashAttention-2. Details in docs/precision.md.
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19