MoE combine traffic cut from 16 to 8.25 bits per value via RHT + S2F6 quantization
snowclipsed · x · 2026-11-14
snowclipsed highlighted a MoE training communication optimization: using a random Hadamard transform plus Blackwell's block-scaled S2F6 PTX instructions, combine-phase traffic drops from 16 to 8.25 bits per value, with fp8 further cutting bytes. RHT and quantization (one scale factor per 32 values) run in the epilogue of the w2 grouped GEMM; the receiving side dequantizes and aggregates on the fly before inverting the transform, shrinking the combine tail and speeding up the MoE layer. The thread also praises dequantization hiding, pipelining, and combine/w2 overlap.
Related event: MoE Training Communication Slashed via 8-bit Quantization and Fused Kernels(4 posts)→
More from Infra
- Redditor predicts Chinese AI chips will flood the market as Nvidia stumbles — Apart_Tumbleweed_303 · 2026-11-14
- FAA issues ground stop, likened to "Microsoft yesterday" systemwide outage — Kyrannio · 2026-11-14
- GAO: 76% of US air traffic control systems unsustainable, still on floppies and Win95 — MartinGTobias · 2026-11-14
- Report claims Jensen Huang resigns as Nvidia CEO; CFO Colette Kress named interim CEO — pdamodaran · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Drogue raises $20M to power floating data centers with the ocean's heat — arian_ghashghai · 2026-11-14