Quantized all-to-all: NVFP4 dispatch plus RHT+S2F6 8-bit combine to fix MoE comms bottlenecks
snowclipsed · x · 2026-11-14
- Discussion of quantized communications in low-precision training: dispatch tokens in NVFP4 and compute WGRAD from quantized (not high-precision) forward activations, cutting NVLink traffic enough to relieve dispatch bottlenecks where MoE GEMMs stall waiting for tokens.
- Combine-stage output activations must stay in higher precision (BF16 by default) to avoid hurting model quality; MXFP8 quantization of combine activations was tried but impaired performance due to poor outlier representation.
- The fix: a 32-dimensional random Hadamard transform (RHT) followed by block-scaled S2F6 (8-bit) quantization shows no measurable accuracy degradation in training; overlapping with w2 plus fp8 further reduces bytes transferred.
Related event: MoE Training Communication Slashed via 8-bit Quantization and Fused Kernels(4 posts)→
More from Infra
- Redditor predicts Chinese AI chips will flood the market as Nvidia stumbles — Apart_Tumbleweed_303 · 2026-11-14
- GAO: 76% of US air traffic control systems unsustainable, still on floppies and Win95 — MartinGTobias · 2026-11-14
- MoE combine traffic cut from 16 to 8.25 bits per value via RHT + S2F6 quantization — snowclipsed · 2026-11-14
- Report claims Jensen Huang resigns as Nvidia CEO; CFO Colette Kress named interim CEO — pdamodaran · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Drogue raises $20M to power floating data centers with the ocean's heat — arian_ghashghai · 2026-11-14