CPU-Only Inference Engine Project Zero v0.2.0 Beats llama.cpp on DeepSeek-V2 MoE
shifu_legend · reddit · 2026-10-24
Developer shifulegend shipped v0.2.0 of Project Zero, a CPU-first LLM inference engine written in C with no ML framework dependency. One binary now runs BitNet/Falcon ternary models, Ternary-Bonsai-27B, dense GGUF, DeepSeek-V2 and Qwen3 MoE, and Mamba-1 — no GPU or Python required.
Performance
- DeepSeek-V2-Lite-Chat Q4KS went from 6.3-7.3 tok/s to 11.7-12.6 tok/s on a 4-vCPU AVX-512 Xeon, beating recorded llama.cpp figures of 9.4-10.5 tok/s; resident memory dropped 12.0GB → 10.6GB.
- Gains came from fused expert FFNs, integer kernels with 8-bit activations, dynamic expert scheduling, and keeping weights quantized.
New features
- --parallel N batches concurrent chats into a shared forward pass: SmolLM2-135M aggregate throughput 45→101 tok/s with 4 clients; SmolLM2-1.7B 12.7→29.8 tok/s.
- Built-in browser chat UI, OpenAI-compatible API, root-free checksum-verified installer, guided setup, grammar-constrained JSON output, speculative decoding, LoRA adapters, int8/int4 or disk-backed KV cache.
- Startup cut from 20-70s to 1-8s.
The author invites benchmark reports across hardware, especially help closing the remaining dense Q4K gap vs llama.cpp.
More from coding & agent
- Moonshot rolls out Kimi K3.1 Pro: 3T params, 1M context, plus 2-3x faster Highspeed variant — cedric_chee · 2026-10-24
- LangChain Academy: agent context engineering starts with retrieval — search small, offload, delegate — LangChain · 2026-10-24
- Autodesk experiments with multiple ChatGPT agents working simultaneously in Fusion — burhop · 2026-10-24
- Happenstance + Claude connector: two prompts that turn your network into a lead list — rubenhassid · 2026-10-24
- Omabib: an open-source Zotero alternative with built-in MCP for Claude Code and Codex — GolinoHudson · 2026-10-24
- OpenResearch integrates Trackio: agents log experiments and read curves to plan next runs — Gradio · 2026-10-24