Running DeepSeek v4 Flash Locally on CPU: RTX 4090 + Tesla P40 Setup
DigiDecode_ · reddit · 2026-08-09
The author shares a detailed experience of deploying and running DeepSeek v4 Flash 0731 locally using a mixed hardware setup (RTX 4090 + Tesla P40 + 128GB RAM).
Deployment Details & Challenges:
- Quantization: Tested Unsloth 4bit KXL (144GB), but lacked memory for DSpark MTP. Switched to IQ4XS quant (127GB) to fit both the model and MTP.
- Inference Speed: Achieved 3 tokens/sec generation and 30 tokens/sec prompt processing with MTP enabled at a 5k+ context size.
- Hardware Allocation: Since llama.cpp doesn't currently support tensor splitting for this model, manual layer-by-layer assignment is required. The embed and output layers must remain on the RTX 4090 (due to unsupported ops for Gated Delta Net on the Tesla P40), while other layers are split between the P40 and CPU.
- Optimization: The ordering of the -dev flag significantly impacts performance, with CUDA0 (RTX 4090) placed first drastically boosting prompt processing speeds.
More from Infra
- Designing Virtualized Execution Environments for AI Agents: Is Firecracker the Best Foundation? — ankush2324235 · 2026-08-10
- Syracuse PhD Thesis: Scaling Logical Reasoning on GPUs to Break CPU Bottlenecks — moyix · 2026-08-10
- New ComfyUI Node Speeds Up Model Loading by Up to 2× via RAM Pre-reading — Valuable-Subject-274 · 2026-08-10
- Terafab Megaproject: Over 100 Million Sq Ft of Manufacturing Space — SIGKITTEN · 2026-08-10
- Developer Proposes Lighter LLM Inference Libraries Over Monolithic Engines — charles_irl · 2026-08-10
- Two Config Tweaks Boost Ling-3.0-flash INT4 Inference Speed by 85% — AcanthisittaOk1699 · 2026-08-10