Running Kimi K3 on CPU: Custom Q3 Quantization Takes 1.1TB, Hits 4.2 t/s
Fun-Meaning-6474 · reddit · 2026-07-30
A developer team used a custom fork of llama.cpp to quantize Kimi K3 (2.8T total parameters, 50B active) into GGUF format. The Q3KS version is currently working and takes up about 1114.76 GiB on disk.
Hardware & Performance:
- They rented a GPU-less box with an AMD EPYC 9554P (64 cores), 1.5TB DDR5 RAM, and NVMe drives in RAID 0.
- Running with 110 threads, it achieved 4.21 t/s for pp512 (prefill 512 tokens).
Quality Testing:
The team tested the quantized model for text coherence and image understanding. For instance, they inputted a front page of the 1969 NYT moon landing, and the model accurately described the masthead, slogan, date, and headlines without hallucination. They are exploring whether running such massive quantized models on CPUs is a viable alternative to smaller, faster models.
More from Infra
- Unified FP8 in Training and Rollout Speeds Up RL by 16% — joecole · 2026-07-30
- ThunderAgent Engine: 2× Throughput, Near-Linear Multi-Node Scaling for Agent Workflows — togethercompute · 2026-07-30
- ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute · 2026-07-30
- QuixiCore-ROCm Open Source: High-Performance Kernel Library Tuned for AMD MI300x — QuixiAI · 2026-07-30
- QuixiAI Open-Sources SlimServe: Fast Inference for GLM on AMD MI300X — QuixiAI · 2026-07-30
- Discussing RTX 3090 VRAM Temperatures Under LLM Workloads — Whole_Alternative_18 · 2026-07-30