BitRun runs 1-bit LLMs at 127 t/s on M4 Pro CPU, zero compile, after fixing BitNet kernel bugs
The_VibePreneur · reddit · 2026-10-14
A developer released BitRun, a zero-compile runner for 1-bit ternary LLMs with an Ollama-style workflow. To make it work, he fixed several low-level bugs in Microsoft's bitnet.cpp (batch stride clobbering, a hardcoded block size causing infinite @ @ @ loops on ARM NEON, SiLU-vs-squared-ReLU activation mismatch, multi-turn BLAS crashes) and submitted PRs upstream.
Verified pure-CPU speeds: 98–127 tokens/s for Falcon3-1B on Apple M4 Pro, 40 t/s for BitNet b1.58 2.4B, 1350+ t/s for BitNet 70M; 39–45 t/s on an Intel i5-12600K. Includes an OpenAI-compatible REST server via bitrun serve.
Caveat: ternary weights require QAT from scratch, so the catalog is limited to sub-3B models — fine for edge devices and background automation, not for heavy coding.
More from Infra
- Runway partners with Baseten for dedicated enterprise deployment of Gen-4.5 and Aleph 2.0 — runwayml · 2026-10-14
- Data center panel asks 'Are We Screwed?' as Gary Marcus says AI industry may have screwed itself — GaryMarcus · 2026-10-14
- Australia drafts national AI and data center framework for 2027 as foreign money flows in — YvesMulkers · 2026-10-14
- ASML Posts €11.7B Q3 Net Sales, 57.5% Gross Margin — BenBajarin · 2026-10-14
- Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs — Sewon Min · 2026-10-14
- AWS walks through deploying and benchmarking models on SageMaker JumpStart with reusable Python script — AWS ML Blog · 2026-10-14