12.8GB near-lossless Inkling-2-Mini quants let iPhones run Haiku-tier models

teortaxesTex · x · 2026-11-22

New quantized versions of Inkling-2-Mini have dropped, ranging from a 12.8GB near-lossless build down to sub-ternary sizes that fit on an iPhone.

Per the benchmark table: the 12.76GB NVF4+Trellis build matches the untouched BF16 teacher on all 349 pooled benchmarks while saving 32.5% weight memory; 11.18GB and 8.87GB builds retain 98.9–99.3% of 3-task mean with lower KL than Unsloth's 9.14GB UD-Q2KXL GGUF. The more aggressive SQYFR v2 scheme, which jointly trains all 24.46B MoE experts and attention trellis projections, lifts the 5.38GB (1.65 bpp) build to 90% top-1 agreement, 87% GSM8K, and 71.3% strict HumanEval; a 4.72GB build sized for the 12GB iPhone memory ceiling hits 82.5% GSM8K with a 179MB drafter. The poster calls it Haiku 5.5-tier on a phone.

Original post →

More from Infra

Infra channel →