Running Kimi K3 on CPU: Custom Q3 Quantization Takes 1.1TB, Hits 4.2 t/s
Fun-Meaning-6474 · reddit · 2026-07-30
A developer team used a custom fork of llama.cpp to quantize Kimi K3 (2.8T total parameters, 50B active) into GGUF format. The Q3KS version is currently working and takes up about 1114.76 GiB on disk.
Hardware & Performance:
- They rented a GPU-less box with an AMD EPYC 9554P (64 cores), 1.5TB DDR5 RAM, and NVMe drives in RAID 0.
- Running with 110 threads, it achieved 4.21 t/s for pp512 (prefill 512 tokens).
Quality Testing:
The team tested the quantized model for text coherence and image understanding. For instance, they inputted a front page of the 1969 NYT moon landing, and the model accurately described the masthead, slogan, date, and headlines without hallucination. They are exploring whether running such massive quantized models on CPUs is a viable alternative to smaller, faster models.
More from Infra
- Trading 2 RTX 5090s for 4 R9700s: does doubled VRAM pay off? — poofph · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- vLLM brings frontier model local deployment to NVIDIA RTX Spark PCs — vllm_project · 2026-10-16
- NVIDIA signals $2B in new European AI deals at GTC Berlin, totaling ~$10B in a year — CurieuxExplorer · 2026-10-16
- Making Your Data Ready for Agentic AI: trust first, then semantics, then writes — bibryam · 2026-10-16