Colibri: Run 2.8T-parameter MoE models on a 25GB laptop with pure C inference engine
alex_verem · x · 2026-08-16
Developer alexverem introduces Colibri, an open-source inference engine that runs AI models up to 2.8 trillion parameters on a 12-core laptop with 25GB RAM. The project has 24.8k stars on GitHub.
Colibri exploits the sparse activation of MoE models (e.g., GLM-5.2 has 744B parameters but activates only 40B per token), treating storage, RAM, and VRAM as a unified hierarchy, streaming experts from disk on demand and caching hot ones. It's pure C with zero dependencies and runs without a GPU.
Performance: 5.8 tokens/s on 6x RTX 5090 (744B model), 1 token/s on a laptop GPU, and 0.1 token/s on the original 25GB machine.
More from Infra
- CoreWeave's debt costs falling: DDTL spreads tightened from S+9.62% to S+2.25% — BenBajarin · 2026-10-16
- SoftBank lines up US IPOs for SB Energy and AI robotics spinoff Roze, valuations top $50B — nordicinst · 2026-10-16
- TRL reproduces Modal's mixed-precision async RL: plain GRPO collapses, SC+MIS holds 0.72 — QGallouedec · 2026-10-16
- Hugging Face scales Indic pretraining with 2.8T tokens and an open data pipeline — Hugging Face Blog · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16
- 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16