Running Kimi K3 in Pure C99: Low-Config Inference on 8GB RAM

FareedKhan557 · reddit · 2026-08-02

To experiment with massive models locally, a developer built a minimalist inference engine in pure C99 for the 1.56TB Kimi K3 model, successfully running it on a single CPU with just 8GB of RAM.

Core Implementation:

On-Demand Loading: 93% of K3's parameters are routed experts, and only 16 out of 896 experts fire per token. Thus, expert layers are never resident in memory; they are read directly from NVMe on demand.
No Dequantization: The packed 4-bit weights are multiplied straight out of their packed form without a dequantization step. The dense trunk is packed into a single file, streamed layer by layer using known offsets.

Performance:

On a dual EPYC 7763 machine, peak memory usage was limited to 8.24GB, achieving 33 seconds/token.
Allocating 128GB of memory maxed out the speed at 20 seconds/token, with byte-identical output across all memory budgets.

The author admits this isn't practical for serving (requiring 1.7TB of disk and being very slow), but was built purely to understand the MoE architecture by implementing it. The project uses no BLAS or frameworks, achieving this with just six C files and a 176KB binary.

Original post →

Running Kimi K3 in Pure C99: Low-Config Inference on 8GB RAM

More from Infra