Running Kimi K3 in Pure C99: Low-Config Inference on 8GB RAM
FareedKhan557 · reddit · 2026-08-02
To experiment with massive models locally, a developer built a minimalist inference engine in pure C99 for the 1.56TB Kimi K3 model, successfully running it on a single CPU with just 8GB of RAM.
Core Implementation:
- On-Demand Loading: 93% of K3's parameters are routed experts, and only 16 out of 896 experts fire per token. Thus, expert layers are never resident in memory; they are read directly from NVMe on demand.
- No Dequantization: The packed 4-bit weights are multiplied straight out of their packed form without a dequantization step. The dense trunk is packed into a single file, streamed layer by layer using known offsets.
Performance:
- On a dual EPYC 7763 machine, peak memory usage was limited to 8.24GB, achieving 33 seconds/token.
- Allocating 128GB of memory maxed out the speed at 20 seconds/token, with byte-identical output across all memory budgets.
The author admits this isn't practical for serving (requiring 1.7TB of disk and being very slow), but was built purely to understand the MoE architecture by implementing it. The project uses no BLAS or frameworks, achieving this with just six C files and a 176KB binary.
Related event: Open-source engines run Kimi K3 on low-end devices(7 posts)→
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16