Splash 1.7.1 cuts Apple Silicon TTFT 23% via Neural Engine prefill, 2.9x faster shared-prompt chats
oss-jiiim · reddit · 2026-11-23
Open-source local inference engine Splash 1.7.1 ships, with menu bar companion LLMenuBar 0.0.4 supporting all new features.
- Neural Engine prefill cuts TTFT 23% for cold 2K/8K prompts on M6 (Qwen3.8-27B), 6-13% on a 16-core M5 Pro.
- New chats sharing a system prompt and tools see 2.5-2.9x faster TTFT.
- Cached weights load up to 1.54x faster on M5 Max; Flint GGUF loading drops from 11.1s to 3.3s on M5 Pro.
- Eight concurrent requests deliver 13% higher throughput than four on a 64GB M5 Pro.
New server options: --max-concurrent-requests, --sampling-default, --recalibrate-ane. Both tools install via brew.
More from Infra
- Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost — eltokh7 · 2026-11-23
- From-scratch Metal 4 engine runs Inkling-2-Mini at 144 tok/s on M5 Ultra, 128k context in 32GB MacBook — DankpawsDev · 2026-11-23
- LFM3-Omni-2B runs locally on iPhone 17 Pro Max doing agentic web search with native speech generation — fuzhongkai · 2026-11-23
- Custom compiler runs Qwen3.5-0.8B BF16 on one CPU core with 109MB peak RSS via disk reads, seeking baselines — Maui-The-Magificent · 2026-11-23
- Melt_Dem: Every Layer of Inference Optimization Will Collapse Into a Market — const_reborn · 2026-11-23
- Canary tokens in ~2M AI requests expose breaches at 3 providers and broken ZDR promises — ccerrato147 · 2026-11-23