COLIBRI: pure-C zero-dep engine streams 2.8T-param MoE models from disk on consumer hardware
bibryam · x · 2026-09-22
COLIBRI (36.9k GitHub stars) runs frontier MoE models — 744B to 2.8T parameters — on consumer and heterogeneous hardware in pure C with zero engine dependencies, treating storage, RAM, and VRAM as a single inference hierarchy with experts streamed from disk. Nine model families work today, including GLM-5.2/5.3 (744B), Kimi K3 (2.8T), and DeepSeek V4 variants.
The key trick treats MoE weights like JIT code: per token, route experts, deduplicate the batch, serve VRAM/RAM hits while NVMe misses stream in, overlap I/O with compute, and learn hotter weight placement. The author notes it's experimental and not production-ready.
More from Infra
- More inference than training machines means writing will soon beat reading — GregoryDiamos · 2026-09-22
- SayGm Routes Prompts Through Two Attested TEEs With NEAR AI's Confidential Compute — richdotca · 2026-09-22
- Sentdex benchmarks openjev: 169ms on Dell GB10 vs 137ms on RTX 3090 — Sentdex · 2026-09-22
- PyroDash cuts inference cost 96% by having a 4B model call the big one only when needed — jiqizhixin · 2026-09-22
- Underdog's Husky Inference Engine Claims 4.5x Speedup Over MLX, 730 tok/s on MacBook — jimmykoppel · 2026-09-22
- Full-Parameter RL on TPUs: peano_ai Runs 310B MiMo-V2.6 Across 1,000+ TPUs — simonguozirui · 2026-09-22