Colibrì runs a 744B MoE model on a laptop with 25GB RAM — pure C, 32k GitHub stars
量子位 · wechat · 2026-09-26
Colibrì, a pure-C, zero-engine-dependency inference framework with 32k GitHub stars, runs giant MoE models on consumer machines via tiered memory scheduling.
- Dense parts (attention, embeddings, shared experts, 9.9GB at int4 for GLM-5.2) stay in RAM; the 19,456 routed experts (370GB) live on NVMe SSD and are loaded on demand when the Router selects them — described as JIT for model weights.
- Speed comes from LRU caching, expert-heat-based cache priorities, next-layer expert prefetching (71.6% predictable), and optional dual-SSD striping.
- Benchmarks: 12-core CPU + 25GB RAM gives 0.05-0.1 tok/s cold; 128GB CPU-only reaches 1.8 tok/s; 6x RTX 5090 hits 5.8-6.8 tok/s.
- Supports 9 model families including GLM-5.2/5.3, DeepSeek V4 Flash, Inkling (975B), and Kimi K3 (2.8T params, 1.6TB storage, runs from 32GB RAM). Prebuilt binaries for Linux/macOS/Windows; a Web Dashboard visualizes expert placement and hotness.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11