Hybrid CPU/NVMe/GPU Setup Runs DeepSeek's 460GB MXFP4 Weights at 960 tok/s Prefill
HankYeomans · x · 2026-09-25
The author shares a week of local inference experiments: running DeepSeek v4.1's MXFP4 quantized weights (460GB) on a self-built CPU/NVMe/GPU hybrid, boosting prefill from 216 tok/s to 960 tok/s and decode from 13 tok/s to 42 tok/s.
Key finding: the current bottleneck isn't the GPUs but the CPU/memory/NVMe side — only 1 of 20 GPUs reaches 100% utilization while the rest sit at 20-30%. Next, the author plans end-to-end model creation, optimization and ablation from a base checkpoint, arguing the only way to learn is to do.
More from Infra
- How GPUs really run deep learning: a primer on memory hierarchy and optimization — goyal__pramod · 2026-09-25
- VeriTile embeds Triton GPU kernels in Lean, with AI agents writing machine-checked correctness proofs — KaiyuYang4 · 2026-09-25
- Merge Gateway Launches Batch Inference at ~50% of Standard Prices — shensi · 2026-09-25
- New deep-dive article on scaling LLM inference in production — abhijithneil · 2026-09-25
- Lambda engineer shares local inference build rule: 27B models need 24-32GB VRAM — TheZachMueller · 2026-09-25
- Pokee AI demos 36B agent model running fully local on Snapdragon X2 Elite with 32GB RAM — Kyrannio · 2026-09-25