Colibrì runs a 744B MoE model on a laptop with 25GB RAM — pure C, 32k GitHub stars
量子位 · wechat · 2026-09-26
Colibrì, a pure-C, zero-engine-dependency inference framework with 32k GitHub stars, runs giant MoE models on consumer machines via tiered memory scheduling.
- Dense parts (attention, embeddings, shared experts, 9.9GB at int4 for GLM-5.2) stay in RAM; the 19,456 routed experts (370GB) live on NVMe SSD and are loaded on demand when the Router selects them — described as JIT for model weights.
- Speed comes from LRU caching, expert-heat-based cache priorities, next-layer expert prefetching (71.6% predictable), and optional dual-SSD striping.
- Benchmarks: 12-core CPU + 25GB RAM gives 0.05-0.1 tok/s cold; 128GB CPU-only reaches 1.8 tok/s; 6x RTX 5090 hits 5.8-6.8 tok/s.
- Supports 9 model families including GLM-5.2/5.3, DeepSeek V4 Flash, Inkling (975B), and Kimi K3 (2.8T params, 1.6TB storage, runs from 32GB RAM). Prebuilt binaries for Linux/macOS/Windows; a Web Dashboard visualizes expert placement and hotness.
More from Infra
- AI agent cuts Orpheus 3B TTS RTF from 1.03 to 0.87 via TensorRT tuning — TheMoonMidas · 2026-09-27
- TensorFold doubles Qwen 27B decode to 158 tok/s on M3 Ultra with parallel-lane speculative decoding — TheMoonMidas · 2026-09-27
- Skeptical math: DeepSeek unlikely to profit on Huawei hardware vs Liang's 10-month payback target — teortaxesTex · 2026-09-27
- Building a sub-AUD 5k local inference rig: used 3090s vs AMD vs Intel Arc — Smooth-Television-48 · 2026-09-27
- Oracle invokes force majeure as blocked data centers hit $200 billion — generativist · 2026-09-27
- I measured an LLM semantic cache: unsafe queries embed NEARER than safe ones — most systems shouldn't run one — bfeeny · 2026-09-27