744B GLM Runs on 25GB RAM
rohanpaul_ai · x · 2026-07-10
The post demonstrates running a 744B MoE model like GLM-5.2 on a consumer machine with 25GB of RAM, though at a very slow speed.
The core reason is that MoE only activates about 40B parameters per token. The author explains how Colibrì keeps dense tensors in memory while storing routed experts on local NVMe, reducing memory pressure via LRU cache, page cache reuse, and compressed KV cache. The tradeoff is that latency is primarily dictated by disk bandwidth and cache hit rates.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22
- Gavin Baker says Nvidia’s $630B figure would be system revenue, not all Nvidia’s — GavinSBaker · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22