12.8GB near-lossless Inkling-2-Mini quants let iPhones run Haiku-tier models
teortaxesTex · x · 2026-11-22
New quantized versions of Inkling-2-Mini have dropped, ranging from a 12.8GB near-lossless build down to sub-ternary sizes that fit on an iPhone.
Per the benchmark table: the 12.76GB NVF4+Trellis build matches the untouched BF16 teacher on all 349 pooled benchmarks while saving 32.5% weight memory; 11.18GB and 8.87GB builds retain 98.9–99.3% of 3-task mean with lower KL than Unsloth's 9.14GB UD-Q2KXL GGUF. The more aggressive SQYFR v2 scheme, which jointly trains all 24.46B MoE experts and attention trellis projections, lifts the 5.38GB (1.65 bpp) build to 90% top-1 agreement, 87% GSM8K, and 71.3% strict HumanEval; a 4.72GB build sized for the 12GB iPhone memory ceiling hits 82.5% GSM8K with a 179MB drafter. The poster calls it Haiku 5.5-tier on a phone.
More from Infra
- MLX over Vulkan hits 124 tok/s on RTX 5060 Ti, beating llama.cpp CUDA by 25% — SumitGup · 2026-11-22
- Intruder swapped Prime Intellect's hosted DeepSeek for an abliterated copy for 53 hours — mawcopolow · 2026-11-22
- Token economics model projects $1.14T global API spend by 2030, but only 1% of users pay beyond coding — AccBalanced · 2026-11-22
- Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses — PathfinderTactician · 2026-11-22
- llama.cpp Vulkan SCALE fix PR #32735 delivers 2.3-3.2x speedup on Radeon R9700 — pmttyji · 2026-11-22
- $1400 decade-old V100 32GB runs Qwen 27B: Q4_K_M +48% gen speed, Q8_0 +16% prefill — PuzzleheadedArm3732 · 2026-11-22