744B Model Runs on 25GB RAM
techNmak · x · 2026-07-11
This post details an inference method to run a 744B parameter model on just 25GB RAM without a GPU. The trick isn't squeezing the whole model into memory, but ensuring most weights don't stay resident in RAM.
Core Concept
- GLM-5.2 is a MoE model: It has 744B total parameters but only activates about 40B per token.
- Colibrì treats inference as a memory tiering problem:
- Frequently used weights stay in RAM.
- 21,504 routing experts are stored on NVMe.
- The router selects experts on demand and streams them from disk.
- Hot experts are cached in free RAM.
- The full int4 model takes up about 370GB on disk, but the resident memory footprint is only 9.9GB, peaking at around 20GB during chat.
Practical Limitations
- On a base 25GB machine, generation speed is only about 0.05–0.1 tokens/s, because a single token might require around 11GB of disk reads.
- This isn't fast enough for production inference, but it proves that for sparse models, model size and resident memory no longer need to match.
- The runtime logs which experts are activated by prompts and automatically pins the most used ones into spare RAM, meaning repeated use could be faster.
Unresolved Issues
- The project lacks complete benchmarks, so the impact of int4 quantization on model quality is still unclear.
- It's written in pure C with no runtime dependencies, created by a single dev on a 12-core laptop.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
- DeepSeek-V4-Flash tops out at 770 tok/s on one B300 in a vLLM batch test — Moreh · 2026-07-22
- NVIDIA starts shipping 102.4 Tbps Spectrum-6 switches for Vera Rubin AI factories — nvidia · 2026-07-22
- Apple publishes SOC 3 audit reports for Private Cloud Compute — throwfaraway4 · 2026-07-22
- Reddit GPU renters say existing platforms only give you two of three: code, recovery, fair billing — legendpizzasenpai · 2026-07-22
- The Sandboxing Manifesto: Secure Execution Environments for Agents — spirosoik · 2026-07-22