MLX beats llama.cpp 2x on M5 Pro for Inkling Mini, NVFP4 fallback to blame
xlr8harder · x · 2026-11-22
In a thread on the fastest local inference for inkling mini, muradylan shares Apple Silicon numbers: MLX hits 37 tok/s vs 17.7 tok/s for llama.cpp on an M5 Pro — about 2x faster — because the GGUF's NVFP4 layers fall back to CPU on Metal. For bulk jobs, keep reasoning effort low, since high thinks for ages.
Related event: Inkling 2 Mini Runs Smoothly on 24GB M5 Pro, MLX Beats llama.cpp(2 posts)→
More from Infra
- Moore Threads rumored 64GB consumer GPU "Lushan" would double RTX 5090 VRAM — CeFurkan · 2026-11-22
- Sarvam launches Token Factory: Indian GPU inference for open frontier models — itsOmSarraf_ · 2026-11-22
- Researchers find cheap AI inference providers stealing keys and snooping on user requests — The Decoder · 2026-11-22
- Inkling Mini on a 5090: vLLM+DSPark+FP8 KV hits 300-350 tok/s — willcb · 2026-11-22
- Musk: Tesla's data is so vast it keeps less than a millionth of it — Scobleizer · 2026-11-22
- Sovereign AI scorecard: Google Cloud tops index at 3.45, Alibaba Cloud last at 1.21 — aftahi_ai · 2026-11-22