Developer shows how to run full trillion-param LLMs locally on CPU for ~$6,000

On August 27, developer carrigmat posted a series of threads systematically debunking the common belief that trillion-parameter open-source models require $100k-class GPU setups to run, and laid out a complete, tested CPU-only local deployment: a server with dual AMD EPYC, 768GB of DDR5 RAM, and no GPU, running the original (non-distilled) DeepSeek-R1 with Q8 quantization (near-maximum quality), at a total hardware and software cost of about $6000. The post includes a full parts list and download links.

Confirmed

• Measured data: 4-drive NVMe array + 768GB RAM, inference at roughly 2 tok/s via llama-server; the same machine previously ran DeepSeek-R1 Q8 at about 1.x tok/s (the post is truncated, defer to the original thread for exact figures)
• Core principle: weights don't all stay in RAM. Most parameters in modern LLMs are sparsely activated experts accessed only intermittently, so those stay on NVMe; continuously accessed hot weights stay in memory, enabling very large models with limited RAM
• Bandwidth case: a single Gen5 NVMe drive delivers 13GB/s, 16 drives in RAID 0 reach 208GB/s—equivalent to quad-channel DDR5
• Cost case: no need for high-capacity drives; cheap 1TB Gen5 drives with small caches and short write lifespans suffice (the workload is read-dominated), about $3500 for 16 drives
• Engineering feasibility: each PCIe slot can be split into 4 NVMe ports via adapters, solving the motherboard port shortage

Optimization Path and Outlook

• The author admits the official llama.cpp doesn't currently make good use of NVMe arrays and is slow, needing performance tuning; he notes Kimi happens to excel at this kind of work and suggests letting an AI coding agent rework llama.cpp directly
• Specific optimization: refer to llama.cpp PR #25294 and implement speculative prefetching—feed the current layer's activations into the next layer's router to predict and preload expert weights ahead of time, overlapping compute with loading; it runs after a rebuild, and the author is trying to upstream the optimization
• The author claims that with a full 16-drive array, the system could handle 10T (10-trillion) parameter models even by 2027

Why It Matters

This approach compresses the cost of fully running frontier open-source models locally from the $100k range down to a few thousand dollars, with reproducible reasoning and measured data for every link (hardware selection, bandwidth, lifespan, interfaces, software)—directly useful for local inference and privacy-sensitive scenarios.

2026-08-27 ~ 2026-08-27 · 14 related posts

• Running Kimi K3 (2.8T params) locally at full quality is possible — carrigmat · 2026-08-27
• [source] Run full-quality Kimi locally on dual EPYC and 768GB RAM — no GPU needed — carrigmat · 2026-08-27
• Trillion-param models on a PC: hot weights in RAM, experts on NVMe — carrigmat · 2026-08-27
• [source] 16 NVMe drives in RAID 0 match DDR5 bandwidth for LLMs — carrigmat · 2026-08-27
• 16 Gen5 NVMe drives in RAID 0 hit 208GB/s, matching 4-channel DDR5 — carrigmat · 2026-08-27
• No 16 NVMe Ports? Every PCIe Slot Splits Into Four of Them — carrigmat · 2026-08-27
• 16 Cheap 1TB Gen5 Drives Cost About $3,500 — Enough for Local Frontier Models — carrigmat · 2026-08-27
• [source] Real-world test: 4-drive array achieves 2tok/s for LLM — carrigmat · 2026-08-27
• llama.cpp Underuses Your NVMe Array? The Author Lets Kimi Tweak the Code — carrigmat · 2026-08-27
• Using AI code agents to self-optimize llama.cpp — carrigmat · 2026-08-27
• Optimization tip: Speculative prefetching for MoE models — carrigmat · 2026-08-27
• Speculatively Prefetch MoE Experts: Start from llama.cpp PR #25294 — carrigmat · 2026-08-27
• Recompile, Test, Run: Frontier Intelligence Locally, Ready for 10T Models by 2027 — carrigmat · 2026-08-27
• A $6,000 rig to run full, non-distilled DeepSeek-R1 locally — carrigmat · 2026-08-27