DeepSeek-V4.1-Flash quantized to 2-4 bit mixed precision for dual RTX PRO 6000 with 0.1306 KL divergence
TheZachMueller · x · 2026-10-20
A new Hugging Face release quantizes DeepSeek-V4.1-Flash-1017 in a TrellisMX K2 Mix laid out for two RTX PRO 6000s: 2-bit experts, with the 629 most important experts raised to 3- and 4-bit via ShapleyMCG allocation, sized to keep 3M tokens of KV cache, plus 4-bit TrellisMX Engram tables.
Measured mean tokenwise KL divergence is 0.1306 nats (8.6% below plain K2, top-1 agreement 87.6%); a +5 GB sibling variant supports 2M KV. Only KLD has been measured so far and serving is still a test image. The build and all KL measurements ran on an 8x B200 node donated by Lambda.
More from Infra
- Google Cloud london region outage hits dozens of services; Revolut and Lloyds among affected — neal_lathia · 2026-10-20
- NVIDIA's $500B AI infra deal had "no obvious provenance" as PE debt talks stall — mjdramstead · 2026-10-20
- The Local Computing Rebellion: Subscription Fatigue and Forced AI Drive a Shift Back On-Premises — DavidLinthicum · 2026-10-20
- DeepSeek also open-sources deepseek-recipe, replicating official API token encoding logic — deepseek_ai · 2026-10-20
- DeepSeek open-sources DeepInfer, the production inference framework behind DeepSeek-V4.1-Flash — deepseek_ai · 2026-10-20
- Scott Hanselman serves local inference to his whole house from a Surface Laptop Ultra — lee_stott · 2026-10-20