Lossless BF16 packing cuts Qwen3.8-27B memory by 11.4 GiB while boosting speed 20-24%
EitherMarch1255 · reddit · 2026-11-14
A Reddit user detailed a lossless packed weight format (BF16P12) for Qwen3.8-27B: identical weights bit-for-bit, stored in 12.25 bits each instead of 16.
- Key insight: BF16 sign+mantissa bytes are incompressible (7.97 bits entropy) but exponent bytes are highly local (2.68 bits), so weights are packed in 256-weight blocks sharing a base exponent with a 4-bit code; outliers (1 in 17,000) are listed in block headers
- Only the 2.37 GiB token embedding stays BF16; 50.88 GiB of matrices shrink to 39.51 GiB
- On dual RTX PRO 6000 Blackwell cards with llama.cpp: 29.5→36.7 tok/s single card, 48.8→58.6 tok/s dual; combined with the model's MTP speculative decoding, best prompts hit 91 tok/s
- No full unpacking: for 1–4 tokens/step the multiply reads packed blocks and rebuilds weights in registers
- Freed VRAM converts directly into longer chats (KV cache is 64 KiB/token); the author also ships an "uncensored" variant as a 10 MB per-request add-on instead of a second 52 GB model
More from Infra
- Intruder swapped Prime Intellect's hosted DeepSeek for an abliterated copy for 53 hours — mawcopolow · 2026-11-22
- Token economics model projects $1.14T global API spend by 2030, but only 1% of users pay beyond coding — AccBalanced · 2026-11-22
- Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses — PathfinderTactician · 2026-11-22
- llama.cpp Vulkan SCALE fix PR #32735 delivers 2.3-3.2x speedup on Radeon R9700 — pmttyji · 2026-11-22
- $1400 decade-old V100 32GB runs Qwen 27B: Q4_K_M +48% gen speed, Q8_0 +16% prefill — PuzzleheadedArm3732 · 2026-11-22
- Inference Stack Launches One-Command Self-Hosted LLM Serving with vLLM, LiteLLM and llm-d — AccBalanced · 2026-11-22