A BitNet kernel looked 29× faster, but DRAM bandwidth cut the real gain to 6–10%
shifu_legend · reddit · 2026-07-25
A developer reports spending two weeks on a ternary-matrix kernel that looked 29× faster in isolation, only to discover the real end-to-end gain is likely just 6–10%.
Key takeaways:
- The project is a C99 inference engine for BitNet-style ternary models, built from scratch with gcc and make.
- The new AVX-512BW kernel benchmarked at 74.6 Gop/s versus a 2.5 Gop/s scalar baseline.
- But on the Xeon test box, BitNet decode was already running at about 95% of DRAM bandwidth, so the workload is memory-bound rather than compute-bound.
- The model still works: BitNet b1.58-2B-4T reaches 36 tok/s on 4 CPU threads with no GPU, and the engine also supports regular GGUF dense models.
The post is basically a reminder to profile the right layer of the stack before celebrating headline kernel numbers.
More from Infra
- AMD marks 20 years since ATI deal that paired CPUs and GPUs — DavidBennett__ · 2026-07-25
- A one-column type mismatch broke every production OAuth login on a remote MCP server — Capedcrusader1923 · 2026-07-25
- AI agents may need payment insurance before users trust them with real budgets — danielbaker06072001 · 2026-07-25
- SkyPilot hits 10,000 GitHub stars while powering AI infra across 25+ clouds — skypilot_org · 2026-07-25
- DIY Pi cluster claims a $2,500 path around $15,000 enterprise servers — brianrkelly · 2026-07-25
- A shadow server can virtual-patch legacy power systems that cannot be fixed directly — lauriewired · 2026-07-25