M5 Ultra LLM tip: set prefill step to 8192 for up to 45% faster throughput
bakawolf123 · reddit · 2026-09-25
A practical tuning tip for running LLMs locally on the M5 Ultra: set --prefill-step-size 8192. The step size affects both prefill performance and the drafter fetching logits for MTP (also affects dflash; mlx-vlm needs a small patch for chunked prefill with dflash).
The value must be large enough to fill all cores but not so large it adds extra dispatches.
Benchmarks with GLM-flash-4bit + MTP show clear gains at 8k vs 2048: on 131k prompt tokens, prompttps 735.5 vs 623.6 and generationtps 71.1 vs 50.8 (185s vs 220s total); gains at 32k context are even larger (1056 vs 860 prompttps).
Caveat: some engines like omlx support adaptive step size — qwen3.8-flash-next starting at 2048 on omlx matches 8k performance at 64k+ context and even runs 10% better on smaller contexts — but that's not universal.
More from Infra
- Goldman: hyperscaler capex to grow 54% next year to $1.2 trillion — firstadopter · 2026-09-25
- Fine-tuning a 194M GLiNER2 dataset tagger on HF Jobs costs $1.50, lifting accuracy 10% to 69% — iamrobotbear · 2026-09-25
- A GB300 rack costs $5M at 1,580 kg — $3,165/kg, fentanyl-grade value density — StewartalsopIII · 2026-09-25
- Why GPUs need philox, not xorshift: parallel RNG in AI training explained — abhi9u · 2026-09-25
- Open-source AI System Design guide walks through LLM inference, RAG and multi-agent systems step by step — blaizedsouza · 2026-09-25
- Yoav Goldberg: real inference engineering is KV cache, interconnect, load balancing, batching — yoavgo · 2026-09-25