$6,000 CPU-only server runs trillion-parameter open models locally

On August 27, developer carrigmat posted a systematic rebuttal of the belief that trillion-parameter open models require $100k GPU rigs, presenting a complete, tested CPU-only deployment: a dual AMD EPYC server with 768GB DDR5 and no GPU running original (non-distilled) DeepSeek-R1 at Q8 quantization for roughly $6,000 total, with full parts list and download links; the same quantization strategy applies to the 2.8T-parameter Kimi K3.

Confirmed

• Benchmarks: 4-drive NVMe array + 768GB RAM achieved about 2 tok/s with llama-server; an earlier run of DeepSeek-R1 Q8 on the same machine hit roughly 1.x tok/s
• Core principle: most parameters are sparsely-activated experts kept on NVMe and accessed intermittently, while hot weights stay in RAM
• Bandwidth math: one Gen5 NVMe drive delivers 13GB/s; 16 drives in RAID 0 reach 208GB/s, equivalent to quad-channel DDR5
• Cost: cheap 1TB Gen5 drives with small caches and short write endurance suffice since the workload is read-mostly; 16 drives cost about $3,500
• Endurance and connectors: weights are written once then read-only, and SSD read endurance is effectively unlimited; each PCIe slot can be bifurcated into 4 NVMe ports

Optimization path

• carrigmat acknowledged stock llama.cpp underutilizes the NVMe array and is slow; he suggests letting Kimi—an AI coding agent he considers well-suited to this—tune llama.cpp, and is trying to upstream the changes
• The concrete optimization follows PR #25294: speculative prefetching that feeds the current layer's activations to the next layer's router to overlap compute and loading
• He claims a full 16-drive array could handle 10T-parameter models into 2027

Why it matters

The setup collapses the cost of running frontier open models at full quality from $100k to a few thousand dollars, with reproducible reasoning and measurements at every layer; the practice of using AI coding agents to rewrite inference engines adds a notable new paradigm for local inference and privacy-sensitive self-hosting.

2026-08-27 ~ 2026-08-27 · 14 related posts

• Running Kimi K3 (2.8T params) locally at full quality is possible — carrigmat · 2026-08-27
• [source] Run full-quality Kimi locally on dual EPYC and 768GB RAM — no GPU needed — carrigmat · 2026-08-27
• Trillion-param models on a PC: hot weights in RAM, experts on NVMe — carrigmat · 2026-08-27
• 16 NVMe drives in RAID 0 match DDR5 bandwidth for LLMs — carrigmat · 2026-08-27
• 16 Gen5 NVMe drives in RAID 0 hit 208GB/s, matching 4-channel DDR5 — carrigmat · 2026-08-27
• No 16 NVMe Ports? Every PCIe Slot Splits Into Four of Them — carrigmat · 2026-08-27
• 16 Cheap 1TB Gen5 Drives Cost About $3,500 — Enough for Local Frontier Models — carrigmat · 2026-08-27
• [source] Real-world test: 4-drive array achieves 2tok/s for LLM — carrigmat · 2026-08-27
• llama.cpp Underuses Your NVMe Array? The Author Lets Kimi Tweak the Code — carrigmat · 2026-08-27
• Using AI code agents to self-optimize llama.cpp — carrigmat · 2026-08-27
• Optimization tip: Speculative prefetching for MoE models — carrigmat · 2026-08-27
• Speculatively Prefetch MoE Experts: Start from llama.cpp PR #25294 — carrigmat · 2026-08-27
• Recompile, Test, Run: Frontier Intelligence Locally, Ready for 10T Models by 2027 — carrigmat · 2026-08-27
• [source] A $6,000 rig to run full, non-distilled DeepSeek-R1 locally — carrigmat · 2026-08-27