A BitNet kernel looked 29× faster, but DRAM bandwidth cut the real gain to 6–10%
shifu_legend · reddit · 2026-07-25
A developer reports spending two weeks on a ternary-matrix kernel that looked 29× faster in isolation, only to discover the real end-to-end gain is likely just 6–10%.
Key takeaways:
- The project is a C99 inference engine for BitNet-style ternary models, built from scratch with gcc and make.
- The new AVX-512BW kernel benchmarked at 74.6 Gop/s versus a 2.5 Gop/s scalar baseline.
- But on the Xeon test box, BitNet decode was already running at about 95% of DRAM bandwidth, so the workload is memory-bound rather than compute-bound.
- The model still works: BitNet b1.58-2B-4T reaches 36 tok/s on 4 CPU threads with no GPU, and the engine also supports regular GGUF dense models.
The post is basically a reminder to profile the right layer of the stack before celebrating headline kernel numbers.
More from Infra
- Engineer with thousands of fabricated devices pours cold water on photonic crystal modulator yields — jwt0625 · 2026-10-26
- Anthropic S-1 reveals 48 cents of compute cost per dollar of Q2 revenue — nmasc_ · 2026-10-26
- Persistent AI joins UR network to route private AI inference with encrypted, IP-hidden traffic — 0xSammy · 2026-10-26
- Anthropic's disclosed compute commitments: $114.5B Google, $161.2B Broadcom top the list — BenBajarin · 2026-10-26
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26
- SpaceX AI revenue up 830%, Claude designs 318K antivenom proteins in 64 hours — PeterDiamandis · 2026-10-26