Pure-C CPU LLM engine hits 36 tok/s on Xeon, 1.8× faster than bitnet.cpp
shifu_legend · reddit · 2026-07-23
From-scratch pure-C LLM inference engine beats bitnet.cpp by 1.8× on Xeon
Project Zero is a CPU-only LLM inference engine written in pure C99 with zero external dependencies. The author says it outperforms bitnet.cpp by 1.8× on the same Xeon box, reaching 36.25 tok/s vs 19.33 tok/s on an Intel Xeon Emerald Rapids 4C system.
What it supports
- Microsoft BitNet b1.58-2B-4T with ternary weights and a full REPL/agentic loop
- Qwen Bonsai-27B via direct GGUF loading and zero-copy mmap, with a claim that it avoids OOMs on constrained RAM
Why it is faster
- BitNet weights are packed ternary values
- The kernel avoids unpacking to float and instead uses a 3-instruction VBMI path feeding directly into INT8 VNNI accumulation
- Threading uses C11 atomics with a spin-then-sleep pool to avoid futex syscalls
Benchmark call for help
The author only has data from two machines and is asking the community for x86 CPU benchmark results, especially on older AVX2 chips and high-core Xeon/EPYC systems, for both BitNet and Bonsai-27B.
More from coding & agent
- AXL launches remote MCP server exposing 1,284 sales and marketing APIs to Claude and Codex — EdTech_Padre · 2026-10-27
- GitHub's Thomas Dohmke ships Entire, a Git forge built for agent-human collaboration — carrycooldude · 2026-10-27
- OmarchyTV hits GitHub: Omarchy-style TV launcher with AI TUI and MCP — alexcovo_eth · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Reddit Thread: Keeping AI Agents Grounded in Current Project Information — Few-Ad-1358 · 2026-10-27
- He gave his AI agent a phone number and email — then hired it as social media manager — heyshrutimishra · 2026-10-27