Bonsai Quantization and llama.cpp Upstream Progress
pmttyji · reddit · 2026-07-15
This update focuses on the upstream migration and inference backend support for Bonsai-27B / Ternary-Bonsai-27B.
Upstream Status
- Binary / Q10 has been merged into multiple llama.cpp backends, ready for use on CPU, Metal, CUDA, and Vulkan.
- MLX (1-bit) is still awaiting upstream merge; a temporary fork is required.
Ternary Progress
- Ternary support is gradually migrating into the mainline llama.cpp, though both mainline and fork coexist for now.
- There are currently three GGUF variants requiring distinct loading:
- Q20: group size 128, used in current demos, fork-compatible, but won't load directly on mainline
- Q20g64: group size 64, official mainline format
- PQ20: planned fork format, not yet officially supported
Backend Migration Status
- CPU (ARM NEON + generic) is in mainline
- Metal is in mainline
- Vulkan is pushing upstream
- CUDA is under review
- x86 AVX-512-VNNI is still pending
Performance & Limitations
- The author mentions several related PRs aimed at improving t/s and prefill performance.
- Clear warning: agentic coding is not the strong suit of these 1-bit / ternary models; keep expectations realistic.
- According to the model cards, the binary version retains roughly 89.5% of the average full-precision performance, while the ternary version hits around 94.6%.
More from Infra
- NVIDIA publishes Vera CPU architecture details before AMD’s AI event — ryanshrout · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- HilbertRaum open-sources a fully local AI chat and document analysis app for private use — Vladowski · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22