Pairing M5 Ultra 256GB with M3 Ultra 512GB for disaggregated prefill/decode cuts TTFT from 40s to 18s
Usual_Tackle5892 · reddit · 2026-10-14
The author paired a new M5 Ultra 256GB with an existing M3 Ultra 512GB instead of waiting for an M5 Ultra 512GB: the M5 handles prefill, ships the KV cache over Thunderbolt 5 RDMA (MLX JACCL, 150Gb/s), and the M3 decodes from its 512GB pool.
Single-node numbers (GLM-5.3-Flash 4-bit + MTP, 17.5k-token prompt)
- M5 prefill hits 1,188 tok/s at 8k chunks (2.4x M3) but decode only 1.4x (47.2 vs 32.9 tok/s)
- Profiling shows real matmul components speed up 3-4x, while sparse-attention gather, linear-attention scan and routing kernels are bandwidth-bound at 1.3-1.8x — nearly half of M5 prefill time sits in kernels that can't use its matrix units; a block-sparse kernel is the next big win
Disaggregated result: 17.8s prefill on M5, 358MB cache handed off in 38ms, first token 18.5s total vs 40s before, output identical to the M5's own. GLM-5.3-Flash's KDA layers keep 20KB/token of state, so the M3's spare 300GB can hold 15M cached tokens. Stack: MLX JACCL, mlx-vlm 0.7.6, and a Python prototype serializing cache objects with mx.distributed.send/recv.
More from Infra
- Serverless tool calling could slash agent costs 30x by running 1000 agents per sandbox — madhavsinghal_ · 2026-10-14
- BofA lifts global memory TAM forecast to $2T by 2030, run rate already over $1T — Beth_Kindig · 2026-10-14
- Ayar Labs, Wiwynn and GUC demo integrated optical compute system with live telemetry at OCP — bookwormengr · 2026-10-14
- AnythingLLM ditches Ollama for Lemonade, shrinking engine from 1.8GB to 137MB — jfowers_amd · 2026-10-14
- Gavin Baker concedes some AI deals were fake, cites $1.4T commitments talking point — GavinSBaker · 2026-10-14
- Argon pushes Gemma past 120k tokens/sec on a single A100 with vLLM, a 2.9x speedup — osanseviero · 2026-10-14