744B Model Runs on 25GB RAM
techNmak · x · 2026-07-11
This post details an inference method to run a 744B parameter model on just 25GB RAM without a GPU. The trick isn't squeezing the whole model into memory, but ensuring most weights don't stay resident in RAM.
Core Concept
- GLM-5.2 is a MoE model: It has 744B total parameters but only activates about 40B per token.
- Colibrì treats inference as a memory tiering problem:
- Frequently used weights stay in RAM.
- 21,504 routing experts are stored on NVMe.
- The router selects experts on demand and streams them from disk.
- Hot experts are cached in free RAM.
- The full int4 model takes up about 370GB on disk, but the resident memory footprint is only 9.9GB, peaking at around 20GB during chat.
Practical Limitations
- On a base 25GB machine, generation speed is only about 0.05–0.1 tokens/s, because a single token might require around 11GB of disk reads.
- This isn't fast enough for production inference, but it proves that for sparse models, model size and resident memory no longer need to match.
- The runtime logs which experts are activated by prompts and automatically pins the most used ones into spare RAM, meaning repeated use could be faster.
Unresolved Issues
- The project lacks complete benchmarks, so the impact of int4 quantization on model quality is still unclear.
- It's written in pure C with no runtime dependencies, created by a single dev on a 12-core laptop.
Related event: 744B GLM-5.2 MoE Model Runs Locally on 25GB RAM(5 posts)→
More from Infra
- MiniMax H3 runs fully on a Galaxy S26: 5-second video with audio in ~7 minutes — BidFunny4862 · 2026-10-27
- Unconventional AI acquires inference chip firm NeuReality in 1,000x efficiency bid — NaveenGRao · 2026-10-27
- Musk: robot RAM demand will dwarf everything else, and 100% of superintelligence compute goes to deep space — elonmusk · 2026-10-27
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27
- Wan2GP lands on Pinokio: 1-click AI video on 6GB GPUs — cocktailpeanut · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27