Running 763B DeepSeek on one 32GB card: 50 tok/s and the real cost of 2-bit quant
Ryyn_- · reddit · 2026-11-11
The author ran DeepSeek V4.1 Flash 1017 (763B MoE) as a 2-bit GGUF (232 GiB on disk) on a single RTX PRO 4500 (32GB), 120GB RAM and a Ryzen 9 9950X3D: experts live in RAM, the hottest ones on the GPU, and DeepSeek's DSpark draft model speculatively proposes five tokens.
Key measurements:
- 49.9 tok/s decode with the draft model (median of three runs) vs 37.2 without; the draft doesn't change output
- Versus the original weights, the 2-bit quant shows KL divergence 0.248 and top-1 agreement of 83.5% — the real cost, which comparing one quant against another hides
- Long thinking struggles to finish: an 8000 thinking budget lets both test tasks complete; at 64k with no budget both tasks thought 60,000 tokens and never answered
- Very long thinking can fall into a soft loop of near-identical sentences
- Throughput is RAM-bandwidth bound (DDR5-4800, four sticks)
For 16/24GB cards: the 125B Qwen3.8-Flash-Next runs the same way at 15 tok/s on 16GB + 64GB RAM, 91 tok/s on the 32GB card. The author discloses his MIT-licensed engine fork (FastLocalExperts) and launcher (flelink).
More from Infra
- LeRobot 0.7 streams terabyte robotics datasets from the Hub, cutting data wait to 0.52% of step time — LysandreJik · 2026-11-11
- Gamers built the GPU market, now AI demand prices them out — Reddit venting — ark_forever · 2026-11-11
- CyberGate: vLLM plugin steers LLM refusals per-domain without touching weights — DebtFluffy9219 · 2026-11-11
- Huawei's TSMC 'White Glove' Chip Orders Resurface in Debate over Megaspeed's $4.8B Nvidia Buys — 0xkarasy · 2026-11-11
- SoftBank posts first net loss in 2 years after $3.7B data center writedown — ns123abc · 2026-11-11
- Data Center Backlash Spreads: Community Outrage Kills Upstate NY Project on Abandoned Steel Plant — sebkrier · 2026-11-11