DeepSeek-V4.1-Flash hits ~30 tok/s on a 12GB GPU with expert-skipping tricks
midokino · reddit · 2026-11-21
Reddit user midokino details running DeepSeek-V4.1-Flash (ISTA-DASLab IQ2XS 2.25 bpw, 266GB) on a gaming PC (RTX 5070 12GB, Ryzen 7 9700X, 64GB RAM), open-sourced as a Strata fork. From 5 tok/s out of the box, tuning reached 32.5 tok/s on code and 28 tok/s on mixed tasks, with 519 tok/s prompt reads on 30K tokens. Quality: GPQA Diamond 89.9% (quant card: 90.40, original: 90.91), HumanEval+/EvalPlus 94.5/90.2%. Key optimizations: skipping low-weight experts that would come from SSD (main speedup, source of KLD 0.183 vs 0.057 conservative preset with equal scores); capping the DSpark draft model's VRAM share (code +29%, thinking +22%); closing buffered file handles after load (+7-17% prompt reads); Expert Deferral from KTransformers SOSP '25 (+11%). He's asking for someone with 500GB capacity to measure KLD against the original model.
More from Infra
- Chart: TSMC revenue up 1807% since 2005, no 'cyclical' slowdown in sight — BenBajarin · 2026-11-21
- Inkling 2 Small pretrained on 8.4T tokens in 68 hours, natively in NVFP4 — viksit · 2026-11-21
- Analysts clash over Nvidia: huge demand vs. Chamath's claim its over-earning era may be over — BenBajarin · 2026-11-21
- Running a local MoE model on a $300 decade-old laptop: 10 tok/s with 4GB VRAM — texhnolyze_d · 2026-11-21
- Together AI SVP Simran Arora to headline AI Dev NYC 2026 on AI as its own performance engineer — DeepLearningAI · 2026-11-21
- OpenAI's token efficiency obsession hints launches now wait for serving costs, not research readiness — VraserX · 2026-11-21