PyTorch 2.15 supercharges Mac MPS: MoE decoding up to 6.5x faster, FP8 arrives
LysandreJik · x · 2026-10-20
Hugging Face's PyTorch team details what's new for the MPS backend in PyTorch 2.15, aiming to make the Mac a first-class place to run and train models. The team spent the year profiling real Hub models, finding MPS shortcomings and fixing them upstream, increasingly by replacing MPSGraph with dedicated Metal kernels.
- FP8 on Mac: float8e4m3fn stores in half the space of BF16, with scaled matmul and the full op set available on CUDA/CPU.
- MoE decodes 5-6.5x faster: Qwen3.6-35B-A3B jumps from 9.4 to 61.3 tokens/s on an M3 Ultra via a native grouped matmul decode path, fewer GPU-CPU syncs, and faster top-k routing.
- Less time per token for all LLMs: specialized matrix-vector kernels for single-row linear layers, and 8x faster softmax over the vocabulary.
- More GPU linear algebra: complex LU and triangular solves, faster small-matrix inverses and QR decomposition; new Metal kernels also target expensive audio/video/training workloads, with 2.16 plans already outlined.
More from Infra
- Latency test from 6 cities: Jev beats OpenAI Decisions and Cloudflare Clef — ImaginaryRoyal9725 · 2026-10-20
- Raw data shows OpenAI returned 271 overload errors to a Codex pool in 48 hours — rickasaurus · 2026-10-20
- Researcher claims 4x faster convergence to backprop loss using Flywheel infra — tensorqt · 2026-10-20
- Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
- Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20