From-scratch Metal 4 engine runs Inkling-2-Mini at 144 tok/s on M5 Ultra, 128k context in 32GB MacBook
DankpawsDev · reddit · 2026-11-23
DankAttn (MIT) is an open-source inference engine written from scratch in Objective-C++ and Metal Shading Language for thinkingmachines' Inkling-2-Mini (26.4B MoE, 3-pass looped, native NVFP4/MXFP8 experts, built-in DSpark drafter), with 19GB pre-packed weights.
- M5 Ultra (96GB): 144.5 tok/s single-stream code with DSpark (vs mlx-serve 121.2, llama.cpp 39.5), 3,804 tok/s cold 8k prefill, 259 tok/s across 4 streams; plain decode still trails mlx-serve (90.6 vs 117.1).
- Base M5 MacBook Pro (32GB): 43.6 tok/s greedy, 128k context in 23.3GB RAM with 8.7GB left for macOS.
Key design: no re-quantization — NVFP4/MXFP8 experts kept bit-for-bit (86.23% MMLU, matching SGLang BF16-KV; GGUF re-quant drops to 84.04% and loses DSpark), dense projections packed to W8 (KL=0.00042); bit-exact speculative decoding at any temperature (0.000e+00 diff); zero-copy Split-KV; persistent SSD prefix cache restores 1.5k-8k prefixes in 13-23ms.
More from Infra
- Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost — eltokh7 · 2026-11-23
- Splash 1.7.1 cuts Apple Silicon TTFT 23% via Neural Engine prefill, 2.9x faster shared-prompt chats — oss-jiiim · 2026-11-23
- LFM3-Omni-2B runs locally on iPhone 17 Pro Max doing agentic web search with native speech generation — fuzhongkai · 2026-11-23
- Custom compiler runs Qwen3.5-0.8B BF16 on one CPU core with 109MB peak RSS via disk reads, seeking baselines — Maui-The-Magificent · 2026-11-23
- Melt_Dem: Every Layer of Inference Optimization Will Collapse Into a Market — const_reborn · 2026-11-23
- Canary tokens in ~2M AI requests expose breaches at 3 providers and broken ZDR promises — ccerrato147 · 2026-11-23