Perplexity open-sources Lily, a Metal inference engine for Qwen on Apple Silicon, 35% faster decode than MLX-LM
inductionheads · x · 2026-09-03
Perplexity open-sourced Lily, the local inference engine built for hybrid compute in Perplexity Computer, specialized for Qwen3.6-35B-A3B (MLX affine 4-bit) on Apple Silicon.
Key details:
- Compact Rust runtime managing session state and the generation loop, with custom Metal kernels compiled at runtime
- Optimized around Qwen's exact MoE architecture: prefill/decode kernels, minimal data movement across expert/attention/recurrent paths, tuned tiles and layouts
- 1.23x faster prefill and 1.35x faster decode than MLX-LM on Qwen3.6-35B-A3B Q4
Exposes a minimal OpenAI chat completions subset, greedy decoding only; requires Apple GPU family 10+ (M5 or newer) and macOS 26+. Code, blog, and reproduction steps are public.
Related event: Perplexity Open-Sources Lily, a Local Inference Engine for Apple Silicon(8 posts)→
More from Infra
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- SGLang Summit sponsors jump from 11 to 36 in one week — ying11231 · 2026-10-18
- Oracle CDS hits record 300bps, called first AI trade domino after Barclays sell downgrade — arthurcolle · 2026-10-18
- Agent Coding Workflow Would Cost ~$40k/Month on API Pricing, Dev Reveals — ChrisRackauckas · 2026-10-18
- PS5LM: llama.cpp port turns a jailbroken PS5 into a home LLM server (500+ GitHub stars) — wateriscoding · 2026-10-18