744B GLM Runs on 25GB RAM
rohanpaul_ai · x · 2026-07-10
The post demonstrates running a 744B MoE model like GLM-5.2 on a consumer machine with 25GB of RAM, though at a very slow speed.
The core reason is that MoE only activates about 40B parameters per token. The author explains how Colibrì keeps dense tensors in memory while storing routed experts on local NVMe, reducing memory pressure via LRU cache, page cache reuse, and compressed KV cache. The tradeoff is that latency is primarily dictated by disk bandwidth and cache hit rates.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26
- Ubicloud publishes "Fixing Tool-Call Accuracy in vLLM" — petrusenko_max · 2026-10-26
- Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
- Unsloth Ships MiniMax-M3.1-Flash Quants: 2-bit Retains 91.5% Accuracy, Runs in 150GB RAM — danielhanchen · 2026-10-26