Run full-quality Kimi locally on dual EPYC and 768GB RAM — no GPU needed
carrigmat · x · 2026-08-27
carrigmat lays out a CPU-only hardware recipe for hosting frontier open models locally:
- His base: 768GB DDR5 + dual AMD EPYC, no GPU — demonstrated running DeepSeek-R1 Q8 at 1.5x realtime.
- You don't need to copy it exactly: 128-192GB RAM with good bandwidth suffices; AMD Threadripper with 8-channel DDR5 recommended, plus PCI-E 5.0.
- Premise: Kimi K3 is 2.8T params and would cost $100k+ in RAM/GPUs, but with native experts + Q8 for everything else you can run it locally at full quality.
More from Infra
- Prefix Sliding enables efficient test-time scaling by cutting memory costs — Niklas Muennighoff · 2026-08-27
- Cohere releases Parse 5, claiming best price-performance for enterprise document parsing — irombie · 2026-08-27
- M7 Ultra may feature native FP8, potentially boosting GLM 5.3-flash performance — Brilliant-Hall1387 · 2026-08-27
- ChronoScale announces 50MW NVIDIA GB300 deployment with Microsoft for AI inference — r_jegaa · 2026-08-27
- Engineering Win: mxfp8 x mxfp4 Matmul Outperforms Standard mxfp8 — zephyr_z9 · 2026-08-27
- The Guardian video: everyone hates datacentres — but do we really need them? — nordicinst · 2026-08-27