Pure-C CPU LLM engine hits 36 tok/s on Xeon, 1.8× faster than bitnet.cpp
shifu_legend · reddit · 2026-07-23
From-scratch pure-C LLM inference engine beats bitnet.cpp by 1.8× on Xeon
Project Zero is a CPU-only LLM inference engine written in pure C99 with zero external dependencies. The author says it outperforms bitnet.cpp by 1.8× on the same Xeon box, reaching 36.25 tok/s vs 19.33 tok/s on an Intel Xeon Emerald Rapids 4C system.
What it supports
- Microsoft BitNet b1.58-2B-4T with ternary weights and a full REPL/agentic loop
- Qwen Bonsai-27B via direct GGUF loading and zero-copy mmap, with a claim that it avoids OOMs on constrained RAM
Why it is faster
- BitNet weights are packed ternary values
- The kernel avoids unpacking to float and instead uses a 3-instruction VBMI path feeding directly into INT8 VNNI accumulation
- Threading uses C11 atomics with a spin-then-sleep pool to avoid futex syscalls
Benchmark call for help
The author only has data from two machines and is asking the community for x86 CPU benchmark results, especially on older AVX2 chips and high-core Xeon/EPYC systems, for both BitNet and Bonsai-27B.
More from coding & agent
- Warp's six non-engineering teams all run on Linear and Claude Code — mon__lim · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- Anthropic researcher: 99% of engineers now run swarms of 300+ self-improving agents — AlishaOutridge · 2026-09-11
- Gergely Orosz: Shipping 10x PRs With AI Agents, Sites Fill With Small Regressions — ducha_aiki · 2026-09-11
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11