Open-Source Engine Runs 744B Model on Consumer Hardware
kalyan_kpl · x · 2026-07-13
Introduces Colibri, an open-source, pure C, zero-dependency inference engine: it can run GLM-5.2（744B MoE） on standard consumer machines by streaming MoE experts from NVMe disks via streaming, rather than keeping the entire model resident in memory.
The highlight here isn't necessarily "faster" speeds, but bringing frontier models closer to commodity hardware:
- MoE expert streaming
- speculative decoding
- KV cache compression
- CPU-first execution
- OpenAI-compatible API server
The author summarizes it as a prime example of how algorithmic and system-level innovations can significantly lower the deployment barrier for frontier models, even if inference speeds remain slower than GPU serving.
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27