Benchmark: Inkling 2 Mini runs on a 24GB Mac, MLX 2x faster than llama.cpp
muradylan · x · 2026-11-22
Sharing benchmarks for inkling mini local inference, muradylan shows Thinking Machines' Inkling 2 Mini (26.4B open weights), nominally needing a 32GB Mac, runs fine on a 24GB MacBook Pro M5 Pro. Same model, two runtimes: MLX 3-bit hits 37 tok/s on short prompts, 31.8 at 4K context, 27.7 at 16K; llama.cpp GGUF (Q3KXL 17.7, IQ4XS 16.6) is 2x slower because NVFP4 layers can't compute on Metal and fall back to CPU. Peak memory at 16K context: 16.4GB with MLX 3-bit. Config: reasoning effort low/medium, 512-token prefill chunks, 1000+ output tokens; keep reasoning low for bulk jobs.
Related event: Inkling 2 Mini Runs Smoothly on 24GB M5 Pro, MLX Beats llama.cpp(2 posts)→