NyoomFloat12: Lossless 12-Bit Format Speeds Up Qwen3-32B Inference Up to 2.54x
realDanFu · x · 2026-10-29
@sylviel introduces NyoomFloat12 (NF12), a lossless 12-bit weight compression format for BF16 models. Existing lossless compression saves GPU memory but not time — decoding costs more than it saves. NF12 exploits per-bit entropy structure (near-deterministic sign and exponent bits, repacked into a meta byte) to make decoding cheap enough to accelerate inference. On Qwen3-32B it delivers 1.23× faster token generation and up to 2.54× rollout throughput in shared-prefix rollout workloads, and the freed HBM headroom goes to a larger KV cache — while BF16 saturates at a KV-capacity wall.
More from Infra
- Movistar and Orange activated Starlink coverage after deadly Spain train crash — elonmusk · 2026-10-29
- Microsoft Cloud tops $63B with growth accelerating to 30% YoY as Nadella touts agentic stack — elonmusk · 2026-10-29
- Google's Gary Illyes on fake Googlebot crawlers, honeypots, and the AI scraping boom — gaganghotra_ · 2026-10-29
- Apple's MLX lands kernel optimizations: 2~3x batched decoding, ~2x small-prompt prefill — awnihannun · 2026-10-29
- LessWrong proposal: burn models into fixed-function ASICs to split AI training from deployment and defuse frontier risk — LessWrong 精选 · 2026-10-29
- How one Redditor squeezed Raon-1-mini from 50 to 232 tok/s on a 16GB RTX 5070 Ti — McNiiby · 2026-10-29