BitNet CPU inference is silently wrong: perplexity 6x reference, one-line fix works
matheus-c-feitosa · reddit · 2026-10-20
An independent benchmark (lowbit-atlas) checked low-bit GGUF models on CPU against HF references on ARM Neoverse-N1 and AVX2 x86.
- BitNet b1.58 2B4T: bitnet.cpp main gives wrong output — text looks fine on x86 but perplexity is 6x the reference (113.3 vs 18.79); ARM prints garbage. Cause: SiLU where the model uses ReLU² (#588). A one-line activation fix corrects it; PR #642 bundles x86/ARM fixes. Unpatched, ikllama.cpp with iq2bnr4 is correct on both.
- Ternary-Bonsai 1.7B: correct everywhere it loads. Speed varies up to 34x by file × runtime: ikllama.cpp with -rtr hits 71.4 t/s prompt processing on x86 vs 27.8 for Prism and 2.1 for Q20g64 (no SIMD kernel yet). -rtr is off by default and without it ikllama.cpp is often slowest.
- The "recommended" Q20 file fails to load in all three runtimes; use PQ20 or Q20g64.
All numbers ship with commits, sha256 and rerun scripts, plus a Kaggle/Colab notebook. Analysis was assisted by Claude Code.
More from Infra
- Perplexity's Open-Source Sandbox Platform SPACE Joins Crusoe Cloud's Managed AI Portfolio — AravSrinivas · 2026-10-20
- vLLM Team Details Agent-Assisted GPU Kernel Work: Kimi K3 Hits 1.73x Stock vLLM — lmoroney · 2026-10-20
- Gartner: $3.5T of AI infra spend may face write-downs by 2030, 80% of neoclouds to fold — rohanpaul_ai · 2026-10-20
- Stripe paid $7.5B for a router, but routing is easy — verifying outputs is the real moat — AIGODSEND · 2026-10-20
- Google Cloud upgrades VPC Service Controls with folder-level perimeters for agent traffic, free — rseroter · 2026-10-20
- Amateur researcher fights Stage 4 colon cancer with ML, hits $500-per-run GPU wall — whafa · 2026-10-20