Softmax bug found in all implementations; FlashAttention FP4 fix in CroissAttn
jon_barron · x · 2026-10-25
Jon Barron reports that all existing softmax and cross-entropy implementations — JAX, PyTorch, textbooks — harbor two subtle bugs; a simple rewrite fixes both and eliminates "slingshots" during grokking.
Follow-up: FlashAttention's backward pass has the same zero-sum bug. With FP4 stochastic rounding it's wrong on every draw; the ICLR'27 submission CroissAttn fixes it, unlocking end-to-end FP4 training.
More from Infra
- OpenRouter announces first APAC partnership with Korea's KT Telecom for sovereign models — AccBalanced · 2026-10-28
- Porcupine-like heatsink found inside Calnex Rb/GPS frequency reference during Noctua fan swap — Pytlicek · 2026-10-28
- Unverified: sub-1GB quantized text models reportedly run on phones, VLM rivals Flash Lite class — JosephJacks_ · 2026-10-28
- Critical pre-boot RCE in EDK II PXE driver: one crafted packet hijacks servers before Secure Boot — cyb3rops · 2026-10-28
- Screenless M1 Airs at $30 per 1,000 multicore points spark cheap Mac cluster idea — vincent_koc · 2026-10-28
- RTX 3080 Tensor Core unlock claims 80-87% TFLOPS boost, RTX 50 series blocked — Revolutionalredstone · 2026-10-28