Scratch-built engine beats vendor runtime for ternary 8B models on free ARM cores
Annual_Manner_5901 · reddit · 2026-08-22
The author presents nucleo, a custom inference engine running Bonsai-8B ternary models on Oracle's free 2-core ARM instance. It outperforms the official llama.cpp fork by 14% in decode and 55% in prefill speeds.
Performance Benchmarks:
- 2 cores: nucleo achieves 2.8 tok/s decode, 4.3 tok/s prefill vs. 2.46/2.77 for the official fork.
- 4 cores: nucleo hits 5.3 tok/s decode, 7.7 tok/s prefill vs. 4.84/5.5 for the official fork.
Technical Details:
- Converts models to a 2.125-bpw interleaved format (2.2GB for the 8B model).
- Uses a NEON kernel to accumulate 128-weight blocks in exact int32, ensuring bit-identical output between paths.
- Accuracy scored 19/20 on a deployed exam. The author notes it does not beat mainline llama.cpp on all hardware (e.g., Apple Silicon M3).
More from Infra
- Anthropic hires Google TPU founder Amir Salek to lead chip efforts — dinabass · 2026-08-22
- ComfyUI: Why --cache-classic drastically improves memory management on RTX 5090 — EasternAd8821 · 2026-08-22
- Google Cloud: 4 Practical Pillars of Token Engineering — rseroter · 2026-08-22
- Zhipu's capital surplus vs GPU shortage may lead to US cloud rentals — teortaxesTex · 2026-08-22
- Data centers are digital factories: no local build-out, no frontier sovereignty — yunta_tsai · 2026-08-22
- Nvidia invests in power infrastructure, extending supply chain — BenBajarin · 2026-08-22