Local DeepSeek on GPU+CPU+NVMe: 21x prefill speedup and 95% TTFT reduction after weeks of tuning
HankYeomans · x · 2026-10-12
A developer shares full stats from day one to today for their hybrid GPU+CPU+NVMe local deployment of DeepSeek (v4.1 MXFP4 quantization), after exhausting patches and tweaks:
- Single-stream decode: 14 → 60.75 tok/s (4.34x)
- 4-stream aggregate: 50 → 123.99 tok/s (2.44x)
- 16K prefill: 216 → 4,344 tok/s (20.1x), TTFT down from 75s to 3.69s (−95.1%)
- 32K prefill: 210 → 4,495 tok/s (21.4x), TTFT down from 152s to 7.12s (−95.3%)
Key takeaway: with sustained tuning, the prefill bottleneck of CPU-offload + NVMe tiering can be dramatically mitigated, cutting long-context time-to-first-token from over two minutes to seconds — compelling real-world data for running large models locally on consumer hardware.
More from Infra
- OpenAI chip head Richard Ho on Jalapeno: why Broadcom beat Nvidia and AMD as partner — richardjnieva · 2026-10-12
- SpaceX building $250M GigaBay at Starbase to build up to 1,000 Starships a year — DimaZeniuk · 2026-10-12
- Bittensor's TPN Labs ships first model TPN005, cutting base size ~40% with 32k context intact — const_reborn · 2026-10-12
- Lightning AI launches BYOC: run Studios on your own AWS, GCP, Azure or Slurm infra — LightningAI · 2026-10-12
- Microsoft to scale in-house Maia AI chips to multi-gigawatt inference by 2027 — The Decoder · 2026-10-12
- Microsoft plans huge deployment of its own AI chips to cut costs — pstAsiatech · 2026-10-12