Hybrid CPU/NVMe/GPU Setup Runs DeepSeek's 460GB MXFP4 Weights at 960 tok/s Prefill
HankYeomans · x · 2026-09-25
The author shares a week of local inference experiments: running DeepSeek v4.1's MXFP4 quantized weights (460GB) on a self-built CPU/NVMe/GPU hybrid, boosting prefill from 216 tok/s to 960 tok/s and decode from 13 tok/s to 42 tok/s.
Key finding: the current bottleneck isn't the GPUs but the CPU/memory/NVMe side — only 1 of 20 GPUs reaches 100% utilization while the rest sit at 20-30%. Next, the author plans end-to-end model creation, optimization and ablation from a base checkpoint, arguing the only way to learn is to do.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12