1.58-bit ternary 27B model runs on 12GB cards; CUDA patch adds 28-32% speed
airesearch12 · x · 2026-09-19
PrismML's Ternary Bonsai 2 27B, a 1.58-bit ternary compression of Qwen 3 27B, is impressing on consumer GPUs. One user generated a complete frontend in one shot from a 5.9GB file on an RTX 3060 (41k tokens over 46 minutes, coherent at 77k context, 26 tok/s fresh). A new GitHub project, bonsai-ada-surgery, does kernel-level CUDA surgery on the PrismML llama.cpp fork to push ternary PTQ10 GEMV decode to the DRAM ceiling: bit-exact math, verified by 153/153 backend tests and byte-identical greedy generations, delivering +2832% on an RTX 4070 12GB and 11% on a 2060 Super, with the full 262k window resident.
Related event: PrismML's Ternary Bonsai 2 27B Shrinks to 5.9GB, Retains 98.2% Performance(20 posts)→
More from Infra
- Training-free page-sparse decode in llama.cpp: 1.72× faster 128k decoding on an RTX 5090 — topppggg · 2026-11-16
- llama.cpp offloads CI caches and runners to Hugging Face to survive 35 PRs/day — ggerganov · 2026-11-16
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- ZGX Fury stress-tested: 351 tok/s on Qwen3.8-Flash-Next, offline agents and local RAG demos — Sam Witteveen · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16