Ternary Bonsai-2 vs Qwen3.5-2B on A13/A14/M3: 32-55% faster decode, and why thermal logging matters
leo-k7v · reddit · 2026-11-21
A developer benchmarked PrismML's Ternary-Bonsai-2 (2B/4B, Q20 ternary weights with Hadamard fold) in his own Swift+Metal inference engine against Qwen3.5-2B IQ4NL on three Apple chips.
Key lesson: log thermal state
- The same A14 decoded at 11.7 t/s in the morning and 23.6 t/s in the evening — morning runs sat at "serious" thermal state. Without thermal logging, phone numbers are worthless.
Results
- A14 (nominal): Bonsai2 2B is 22% faster on prefill and 32% faster on decode than Qwen3.5-2B at 4 bits; the 4B peaks at 567MB footprint (mmapped weights excluded) but takes 9s of prefill before the first token.
- A13 (no matrix units): prefill runs 6x slower via the gemv path, hitting serious thermal within one run; decode remains usable.
- M3: ternary 2B decodes 55% faster (94 vs 60.8 t/s).
- Quality: help test 5/5, 6/6 puzzles in 138s, perplexity 24.61 vs 24.66 on upstream llama.cpp — the ternary decode matches upstream.
Optimizations
- Fusing Hadamard into RMSNorm halved dispatches but yielded zero measurable gain — 2B isn't dispatch-bound on these chips.
- Keeping attention tile context sums in registers gave +7% prefill at head dim 256 and +20% at 128/512 on M3; decode unmoved.
Details come from the author's ChatOKF project, a pure-Swift-on-Metal local GGUF client.
More from Infra
- Inkling 2 Mini quantized to 5.4GB/8.9GB, keeping 91-97% performance — teortaxesTex · 2026-11-21
- Open-weight models are less efficient, so their rise may increase chip sales — aronchick · 2026-11-21
- Score centering, rumored OpenAI RL trick, tolerates staleness and may rescue slow chips — teortaxesTex · 2026-11-21
- $31/month quad-channel EPYC + 128GB runs 3-bit Qwen at 128k context — Ne00n · 2026-11-21
- Essay 'The Organic Web' weaves 62 citations to dissect data's physical weight — round · 2026-11-21
- ARK projects Starship marginal launch cost falling to $358/kg by 2027 — mitchdeg · 2026-11-21