Zero-Dependency C99 BitNet Inference Hits 36 tok/s on Intel Xeon CPU
shifu_legend · reddit · 2026-08-09
A developer built a CPU-first inference engine from scratch in pure C99, running BitNet 1.58-bit ternary models natively without Python, CUDA, or BLAS.
Core Tech & Performance:
- Achieves 36.25 tok/s on BitNet b1.58-2B-4T using 4 threads on an Intel Xeon.
- Native ternary SIMD: Weights are packed 4 per byte; custom AVX2/AVX-512 routines with VNNI instructions accumulate directly into integer registers, avoiding float32 unpacking overhead.
- Minimal runtime: Thread pool uses C11 atomics with spin-then-yield backoff for near-zero sync overhead. Compiles to a standalone binary serving an OpenAI-compatible API.
Bottleneck & Takeaways:
- Hits the DRAM bandwidth ceiling. At batch size 1, decode speed is memory-bound. Running at 95% of theoretical memory bandwidth, further compute kernel optimizations won't improve end-to-end latency.
More from Infra
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16
- DeepSeek V4.1 Flash on 4x DGX Spark: weights alone are 511GB, the 494 tok/s claim doesn't fit — Aroochacha · 2026-10-16
- Open-sourced Intel GPU Vulkan inference upgrades: 2x ingestion, 60%+ faster inference — Fun-Aardvark-1143 · 2026-10-16
- ELSFP pain points: >3dB insertion loss, costly PMF, harsh packaging requirements — jwt0625 · 2026-10-16