A BitNet kernel looked 29× faster, but DRAM bandwidth cut the real gain to 6–10%

shifu_legend · reddit · 2026-07-25

A developer reports spending two weeks on a ternary-matrix kernel that looked 29× faster in isolation, only to discover the real end-to-end gain is likely just 6–10%.

Key takeaways:

• The project is a C99 inference engine for BitNet-style ternary models, built from scratch with gcc and make.
• The new AVX-512BW kernel benchmarked at 74.6 Gop/s versus a 2.5 Gop/s scalar baseline.
• But on the Xeon test box, BitNet decode was already running at about 95% of DRAM bandwidth, so the workload is memory-bound rather than compute-bound.
• The model still works: BitNet b1.58-2B-4T reaches 36 tok/s on 4 CPU threads with no GPU, and the engine also supports regular GGUF dense models.

The post is basically a reminder to profile the right layer of the stack before celebrating headline kernel numbers.

Original post →

More from Infra

• LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
• PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
• Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
• RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
• SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
• SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11

Infra channel →