744B Model Runs on 25GB RAM
techNmak · x · 2026-07-11
This post details an inference method to run a 744B parameter model on just 25GB RAM without a GPU. The trick isn't squeezing the whole model into memory, but ensuring most weights don't stay resident in RAM.
Core Concept
- GLM-5.2 is a MoE model: It has 744B total parameters but only activates about 40B per token.
- Colibrì treats inference as a memory tiering problem:
- Frequently used weights stay in RAM.
- 21,504 routing experts are stored on NVMe.
- The router selects experts on demand and streams them from disk.
- Hot experts are cached in free RAM.
- The full int4 model takes up about 370GB on disk, but the resident memory footprint is only 9.9GB, peaking at around 20GB during chat.
Practical Limitations
- On a base 25GB machine, generation speed is only about 0.05–0.1 tokens/s, because a single token might require around 11GB of disk reads.
- This isn't fast enough for production inference, but it proves that for sparse models, model size and resident memory no longer need to match.
- The runtime logs which experts are activated by prompts and automatically pins the most used ones into spare RAM, meaning repeated use could be faster.
Unresolved Issues
- The project lacks complete benchmarks, so the impact of int4 quantization on model quality is still unclear.
- It's written in pure C with no runtime dependencies, created by a single dev on a 12-core laptop.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- 12 KV Cache Reduction Techniques Every AI Engineer Should Understand, Explained — blaizedsouza · 2026-09-11
- The shadow GPU capacity market is formalizing, with Meta selling excess compute to outside buyers — DavidLinthicum · 2026-09-11
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11