Perplexity open-sources Lily, a Rust/Metal inference server mapping Qwen ops to Apple silicon
perplexity_ai · x · 2026-09-03
Perplexity released and open-sourced Lily, designed to treat Apple silicon as a distinct inference platform and map Qwen's operations directly onto its compute and memory architecture.
GitHub repo (pplx-garden/lily, Rust) highlights:
- A Metal inference server for one checkpoint: Qwen3.6-35B-A3B as MLX affine 4-bit weights (group size 64)
- Exposes a minimal subset of the OpenAI chat completions API, always decodes greedily
- Metal kernels compile from source at runtime, no offline shader build
- Requires Apple GPU family 10+ (M5 and newer), macOS 26+, Rust 1.92
- Validates exact architecture and quantization layout at load time, with benchmark reports and reproduction steps
The launch post also recaps the thread's benchmarks: 1.23x prefill and 1.35x decode throughput vs MLX-LM on M5 Max.
Related event: Perplexity Open-Sources Lily, a Local Inference Engine for Apple Silicon(8 posts)→
More from Infra
- CoreWeave co-founders cash out $3.3B in stock as AI credit risks mount — SumitGup · 2026-10-18
- ClickHouse aggregates 100B rows in 0.4s with parallel replicas auto-scaling — Abhishekcur · 2026-10-18
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- SGLang Summit sponsors jump from 11 to 36 in one week — ying11231 · 2026-10-18
- Oracle CDS hits record 300bps, called first AI trade domino after Barclays sell downgrade — arthurcolle · 2026-10-18