744B GLM Runs on 25GB RAM
rohanpaul_ai · x · 2026-07-10
The post demonstrates running a 744B MoE model like GLM-5.2 on a consumer machine with 25GB of RAM, though at a very slow speed.
The core reason is that MoE only activates about 40B parameters per token. The author explains how Colibrì keeps dense tensors in memory while storing routed experts on local NVMe, reducing memory pressure via LRU cache, page cache reuse, and compressed KV cache. The tradeoff is that latency is primarily dictated by disk bandwidth and cache hit rates.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11