Mirai's uzu engine brings speculative decoding to Apple M5, hitting 105 tok/s on Qwen3.6 27B
TheMoonMidas · x · 2026-09-04
Mirai released a speculative decoding implementation in its local inference engine uzu, initially supporting Qwen3.6 27B with Qwen3.8 27B and Muse Glimmer to follow.
- On Apple M5 Max, Qwen3.6 27B 4-bit (Mirai-M) outputs 105 tok/s — nearly 2x MTPLX (55 tok/s) and over 3x llama.cpp (30 tok/s) at comparable quantization.
- Gains vary by domain: 113.84 tok/s on HumanEval and 117.30 tok/s on MATH-500 vs 84 tok/s on MT-Bench, since code is easier to predict than chat.
- The draft model, quantization format, verification algorithm, and GPU kernels are co-designed from the ground up for Apple M5's Neural Accelerators.
- Runs via brew install mirai (14.5 GB checkpoint), with an SDK for Swift, TypeScript, Python, and Rust.
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20