Quantizing training to MXFP8 with stochastic rounding: the accuracy-speed sweet spot
snowclipsed · x · 2026-11-14
Discussion of a training quantization scheme: stochastically quantize weights and activations to MXFP8 and compute gradients with MXFP8 matrix multiplications, claimed as the best accuracy-vs-speed tradeoff — an mxfp8-like SR method. The attached diagram lays out the full pipeline: BF16 activations quantized per-token to NVFP4 (4/6) for the forward GEMM, FP32 master weights quantized per-expert to NVFP4, while gradient paths (DGRAD/WGRAD) use MXFP8 with stochastic rounding, with FP32/BF16/MXFP8/NVFP4 precision boundaries marked. The poster wonders where the fusion boundaries should sit.
Related event: MoE Training Communication Slashed via 8-bit Quantization and Fused Kernels(4 posts)→
More from Infra
- Redditor predicts Chinese AI chips will flood the market as Nvidia stumbles — Apart_Tumbleweed_303 · 2026-11-14
- GAO: 76% of US air traffic control systems unsustainable, still on floppies and Win95 — MartinGTobias · 2026-11-14
- MoE combine traffic cut from 16 to 8.25 bits per value via RHT + S2F6 quantization — snowclipsed · 2026-11-14
- Report claims Jensen Huang resigns as Nvidia CEO; CFO Colette Kress named interim CEO — pdamodaran · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Drogue raises $20M to power floating data centers with the ocean's heat — arian_ghashghai · 2026-11-14