OpenBonsai open-sources a pipeline compressing BF16 models to ~2GB ternary GGUFs
ButtercupLyn100 · reddit · 2026-10-29
A developer released OpenBonsai (Apache-2.0), filling the gap left by Prism ML's Ternary Bonsai: a complete pipeline to compress your own BF16 fine-tunes into ternary GGUFs compatible with Prism's PTQ10 format.
Method
- Signed Hadamard rotations to reduce weight outliers
- Sequential GPTQ to {-1, 0, +1} weights with FP16 group scales (one per 128 weights)
- Calibration on the model's self-generated reasoning traces (inspired by an Intel Labs China paper) plus scale distillation against the original BF16 output distributions
- Embeddings and LM head are also ternary-quantized: Qwen3.5-9B shrinks from 18GB to a 1.997GB GGUF in 5 hours on one RTX 5090
Findings
- Distillation helps substantially: dev NLL 3.48→2.89, ARC-Easy 13/20→19/20
- Calibration data matters: switching to self-generated math/code traces lifted HumanEval from 19/50 to 34/50
- The same pipeline compressed Qwen3-8B to 1.8GB without adapter changes
Limits: Prism's official models still outperform; compression needs a 32GB GPU; inference requires Prism's llama.cpp fork (no upstream llama.cpp or WebGPU yet). The repo includes failed experiments and ablations.
More from Infra
- "Insert coins to continue": meme laments the end of cheap AI tokens — Suspicious_Orchid770 · 2026-10-29
- Skip ComfyUI: run Flux-3-Image-Klein locally with stable-diffusion.cpp in four files — srigi · 2026-10-29
- GPU unlock is pointless for most users: quantized models, decode and gaming see zero speedup — Justify_87 · 2026-10-29
- US AI Power in Three Stages: 10GW+ Nuclear Contracts, Geothermal Live, SMRs Permitting — ingliguori · 2026-10-29
- Revolut self-hosted GLM: 60-70% faster delivery at a tenth of frontier API costs — AI Engineer · 2026-10-29
- Jarvislabs to Bring 1,024 NVIDIA B200 GPUs Online in December, Reservations Open — amaarora · 2026-10-29