Lossless MXFP4 scale compression frees 50.2 GiB across 8 GPUs running Kimi-K3.1-Pro
AlpinDale · x · 2026-11-12
AlpinDale opened a PR to dphnAI/sonar adding lossless compression of MXFP4 expert scales for Kimi-K3.1-Pro-NF12. Benchmarked on 8×B300 with 131K input / 2K output tokens and no speculative decoding: model memory drops from 193.80 to 187.52 GiB per GPU (50.2 GiB total across 8), freeable KV memory per GPU rises 12.5% (50.76→57.11 GiB), at a cost of 2.3–3.3% output throughput and 3.6–4.8% per-token latency.
More from Infra
- Six techniques that decide LLM serving performance, explained — techNmak · 2026-11-12
- MiniMax H3 Needs 50 GB — Here's How It Runs on a 12 GB GPU — fa6637 · 2026-11-12
- PISA: Prompt-Adaptive Cost-Aware LLM Routing Tops Four Routing Benchmarks — Saul Urso · 2026-11-12
- Join ordering, Kaplan-Meier decode estimates and trie attention yield 10x analytics inference speedup — charles_irl · 2026-11-12
- Bernstein's Rasgon: Nvidia's growth comes down to Jensen Huang being a visionary — ceciletamura · 2026-11-12
- Local AI Daily: DeepSeek-V4.1-Flash hits 90 tok/s on 4× DGX Spark; Runway open-sources Praxis-1 — AIFlow_ML · 2026-11-12