Why the BF16-to-12-bit compression trick won't extend to FP8 or FP32 weights
AlpinDale · x · 2026-11-07
- FP8 resists the trick: most FP8 quants use per-block/per-channel scaling to stretch each block across the full E4M3 range (amax capped at 448), spreading stored exponents instead of clustering like BF16 weights — so escape rates would be far higher. E4M3's NaN pattern (S.1111.111) and subnormals would also need escape handling.
- FP32 is pointless: nobody trains LLMs in FP32 anymore; usable weights are upcast from BF16 with the low 16 mantissa bits all zero, so encoding would just compress back to BF16 — recreating NF12.
- Takeaway: BF16 is already near-lossless; the method could theoretically apply to any bit-width, but writing efficient inference kernels for most is a mathematical nightmare.
Related event: Developer open-sources nf12: lossless 12-bit compression for BF16 models(10 posts)→
More from Infra
- US AI scale now hinges on coordinating power, components, agentic workflows and humanoid pilots — ingliguori · 2026-11-07
- Fudan's D3D diffusion drafter enables async speculative decoding, up to 7.48x speedup on Qwen3 — pmttyji · 2026-11-07
- Study: data centers have only marginal impact on residential electricity prices — TinfoilTricorn · 2026-11-07
- YouJiacheng: post-training needs high tps for long-tail rollouts — YouJiacheng · 2026-11-07
- vLLM contributor: lossless KV cache quantization viable but not worth it — AlpinDale · 2026-11-07
- NVIDIA's first co-packaged optics Ethernet switch, Spectrum-X Photonics, enters full production — TinfoilTricorn · 2026-11-07