From-scratch Metal 4 engine runs Inkling-2-Mini at 144 tok/s on M5 Ultra, 128k context in 32GB MacBook

DankpawsDev · reddit · 2026-11-23

DankAttn (MIT) is an open-source inference engine written from scratch in Objective-C++ and Metal Shading Language for thinkingmachines' Inkling-2-Mini (26.4B MoE, 3-pass looped, native NVFP4/MXFP8 experts, built-in DSpark drafter), with 19GB pre-packed weights.

Key design: no re-quantization — NVFP4/MXFP8 experts kept bit-for-bit (86.23% MMLU, matching SGLang BF16-KV; GGUF re-quant drops to 84.04% and loses DSpark), dense projections packed to W8 (KL=0.00042); bit-exact speculative decoding at any temperature (0.000e+00 diff); zero-copy Split-KV; persistent SSD prefix cache restores 1.5k-8k prefixes in 13-23ms.

Original post →

More from Infra

Infra channel →