Ternary Bonsai 2 27B at 1.75bpw fits an 8GB GPU, hits 93.3% accuracy in audiobook speaker-attribution test
autonoma_2042 · reddit · 2026-09-19
A detailed hands-on evaluation pits PrismML's Ternary-Bonsai-2-27B (ternary {−1,0,+1} weights at 1.75bpw, 5.95GB, dense Qwen3.8-27B base with hybrid Gated DeltaNet attention) against Gemma 26B-A4B MoE for speaker attribution in an audiobook pipeline, running on an 8GB NVIDIA T1000.
Key findings and pitfalls:
- Requires the PrismML llama.cpp fork (stock llama.cpp rejects PTQ10); Turing TU117 prefills at 18 tok/s.
- Spilling even one layer to CPU causes a 2.6× slowdown (6.5 → 2.5 tok/s): generic CPU kernels with no AVX path, plus fused DeltaNet ops get disabled for every layer.
- Final config -ngl all -b 2048 -ub 256 -ctk q80 -ctv q80 fits in 6.6GB; quotations per request were halved to 16 to avoid 8192-context overflows costing 13 min each.
- Scores 93.3% (448/480) on hand-corrected ground truth, but suffers narrator-identity drift and adjacent-turn swaps in rapid dialogue.
Verdict: the 5.9GB ternary dense model is viable on a single 8GB card, though long-context reasoning and speaker consistency lag. The methodology and exact flags are directly reusable for local-deployment tinkering.
More from Infra
- Observer v3: open-source agent that watches your screen with local LLMs — Roy3838 · 2026-09-19
- YuE2 BF16 runs on 12GB VRAM and sounds better, plus the fix for pagefile thrashing — lazyspock · 2026-09-19
- Running AI training on unsupported AMD MI50s: patched ROCm and a Vulkan training stack — Savantskie1 · 2026-09-19
- Inco Splash hits 144 tok/s on Qwen3.8-27B M5 Max, 3x faster than Ollama — ResearchCrafty1804 · 2026-09-19
- macOS 27 ships with mlx_whisper built in, letting agents transcribe video locally — vista8 · 2026-09-19
- Nvidia-backed Nscale files for NYSE IPO, reveals $103.4B contract value — IanAndrewsDC · 2026-09-19