oMLX 0.8 turns NVIDIA GPUs and DGX Spark into prefill accelerators for Mac, up to 5.3x faster

pcuenq · x · 2026-11-16

oMLX 0.8.0rc1 adds a Prefill Accelerator: an NVIDIA GPU PC, DGX Spark, or second Mac runs the first layers of prompt prefill while your Mac keeps decoding with the KV cache. No RDMA or Thunderbolt needed — just LAN, and one docker run on the GPU side. On a 64K prompt with M3 Ultra 512GB: Qwen3.8 27B jumps from 352 to 1,062 tok/s with DGX Spark (+202%) and to 1,879 tok/s with 2x RTX A6000 (+434%). Auto mode splits layers based on measured speeds.

Original post →

More from Infra

Infra channel →