Disk-based streaming inference solution for GLM-5.2

karminski3 · x · 2026-07-14

The focus here isn't the model's capability, but how to run a massive MoE model with just 25GB of RAM.

The author explains the intuitive impossibility: GLM-5.2 has 744B total parameters and 40B active parameters. Even counting just the active part, 8bit/4bit quantization creates immense memory pressure, and reading disk for every token would bottleneck speed on NVMe bandwidth.

Their approach splits the model:

Resident in memory: Dense parts like attention, shared experts, and embeddings (17B, 9.9GB)
On-demand loading: 21,504 routed experts (370GB) stored entirely on disk, streamed during inference
Cache strategy: per-layer LRU + hot spot pinning + system page cache acting as L2
Prefetch optimization: Experimental router-lookahead (PILOT) uses the current layer's state to predict the next layer's routing, hitting 71.6% accuracy
Speculative decoding: Combined with GLM's built-in MTP, achieving a 39–59% acceptance rate, outputting 2.2–2.8 tokens per forward pass

Conclusion: If you lack GPUs but have a strong NVMe/disk array, this framework is worth trying.

Original post →

Disk-based streaming inference solution for GLM-5.2

More from Infra