Disk-based streaming inference solution for GLM-5.2
karminski3 · x · 2026-07-14
The focus here isn't the model's capability, but how to run a massive MoE model with just 25GB of RAM.
The author explains the intuitive impossibility: GLM-5.2 has 744B total parameters and 40B active parameters. Even counting just the active part, 8bit/4bit quantization creates immense memory pressure, and reading disk for every token would bottleneck speed on NVMe bandwidth.
Their approach splits the model:
- Resident in memory: Dense parts like attention, shared experts, and embeddings (17B, 9.9GB)
- On-demand loading: 21,504 routed experts (370GB) stored entirely on disk, streamed during inference
- Cache strategy: per-layer LRU + hot spot pinning + system page cache acting as L2
- Prefetch optimization: Experimental router-lookahead (PILOT) uses the current layer's state to predict the next layer's routing, hitting 71.6% accuracy
- Speculative decoding: Combined with GLM's built-in MTP, achieving a 39–59% acceptance rate, outputting 2.2–2.8 tokens per forward pass
Conclusion: If you lack GPUs but have a strong NVMe/disk array, this framework is worth trying.
Related event: COLIBRI Runs 744B GLM-5.2 Model on 25GB RAM Without GPU(3 posts)→
More from Infra
- Nvidia Is Now Core to Every Major Robotaxi Stack at Commercial Scale — pdamodaran · 2026-09-11
- 12 KV Cache Reduction Techniques Every AI Engineer Should Understand, Explained — blaizedsouza · 2026-09-11
- The shadow GPU capacity market is formalizing, with Meta selling excess compute to outside buyers — DavidLinthicum · 2026-09-11
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11