Bonsai Compresses Qwen 27B from 55.6GB to 5.9GB, Retains 98.2% Benchmark Performance
nicolechirps · x · 2026-10-30
PrismML's Bonsai compression shrank Qwen3.8-27B from 55.6GB to 5.9GB (89.4% reduction) while retaining 98.2% aggregate benchmark performance across 20 evaluations, up from 95% retention in the previous 27B version. The technique is ternary quantization: weights use only -1, 0, +1 plus scaling factors.
A quoted post adds real-world numbers: Qwen3.8-35B-A3B at Q4KM runs on a 12GB RTX 3060 with 16GB RAM via llama.cpp at 50 tokens/s generation and 550 tokens/s prompt processing, with 262K context, Flash Attention, and Q8 KV cache — feasible because the MoE architecture activates only 3B of 35B parameters per step.
More from Infra
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Quantum startups raised $4.9B in VC but sold only $1.9B — a field guide maps all four markets — antavedissian · 2026-10-30
- DeepSeek V4.1 on a single RTX 5090: 4000+ tok/s prefill, 7-24 tok/s decode via NVMe streaming — Front_Eagle739 · 2026-10-30
- Cohere to invest over C$4 billion in Canada, creating 1,500+ jobs by 2031 — cohere · 2026-10-30
- River API Now Trains LLMs With Any Custom PyTorch Loss, Compiled and Run at Scale on Its GPUs — ibab · 2026-10-30
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30