Perplexity open-sources Lily, a Metal inference engine for Qwen on Apple Silicon, 35% faster decode than MLX-LM
inductionheads · x · 2026-09-03
Perplexity open-sourced Lily, the local inference engine built for hybrid compute in Perplexity Computer, specialized for Qwen3.6-35B-A3B (MLX affine 4-bit) on Apple Silicon.
Key details:
- Compact Rust runtime managing session state and the generation loop, with custom Metal kernels compiled at runtime
- Optimized around Qwen's exact MoE architecture: prefill/decode kernels, minimal data movement across expert/attention/recurrent paths, tuned tiles and layouts
- 1.23x faster prefill and 1.35x faster decode than MLX-LM on Qwen3.6-35B-A3B Q4
Exposes a minimal OpenAI chat completions subset, greedy decoding only; requires Apple GPU family 10+ (M5 or newer) and macOS 26+. Code, blog, and reproduction steps are public.
More from Infra
- Reddit: Picking the Best Chat Model for a 3090 Ti Local AI Butler — MarcusAurelius68 · 2026-09-03
- MTP vs MTP+Ngram on Qwen3.8 Flash: 10% Speed but 3x Token Usage — esw123 · 2026-09-03
- Perplexity Computer demos fully local operation on NVIDIA DGX Spark — chrmanning · 2026-09-03
- llama.cpp deprecates --chat-template-kwargs, reasoning-preserve now on by default — Bulky-Priority6824 · 2026-09-03
- Agentic API adds a stateful layer in front of vLLM for open-model agent runtimes — techNmak · 2026-09-03
- Google's Gemini 3.8 Flash 'works harder' but may burn more tokens at same pricing — The Verge AI · 2026-09-03