Ternary Bonsai 2 fully working on ik_llama.cpp: native CUDA offload plus fast CPU via -rtr
Any_Bus9205 · reddit · 2026-10-31
A Reddit guide details running PrismML's ternary Ternary Bonsai 2 (27B) on ikllama.cpp. Upstream llama.cpp's QK=256 block size (TQ20) vs Prism's QK=128 (PQ20) previously broke or bottlenecked CUDA offload due to Hadamard sign tensor handling. Latest commits fix everything:
- Lossless repack (PR #2694): a built-in script converts upstream TQ20/TQ10 GGUFs to PQ20 in seconds with exact trits preserved.
- Native CUDA full offload (PRs #2691 & #2747): merged PQ20 kernels let you offload all layers with -ngl 99, no token corruption or k-quant fallback.
- CPU speedup: the PQ20R8 8-row interleaved format plus -rtr 1 (runtime repack) dramatically accelerates prompt processing.
Full commands and recommended sampling params (temp 0.7 / top-p 0.95 / top-k 20) included; credit to Iwan Kawrakow and contributors.
More from Infra
- NASA JPL stress-tests CXMT memory down to -100°C, unit stays online — jwt0625 · 2026-11-01
- Kaggle Resource Usage Up 2-3x as Agents Flood In for ARC Competitions — GregKamradt · 2026-11-01
- Why AI data centers in space are not practical: cooling, radiation and maintenance — johnseach · 2026-11-01
- Dev runs 10-12 parallel coding sessions; says decision-making becomes the bottleneck — robleclerc · 2026-11-01
- Workshop: Build a media agent on CPU, call image and TTS models as MCP tools — Al_Grigor · 2026-11-01
- Amazon reportedly junking 2-3-year-old H100/H200 GPUs while books still assume 6-year depreciation — SumitGup · 2026-10-31