JITEngine synthesizes per-Mac LLM inference engines, up to 2.7x faster
CShorten30 · x · 2026-11-05
The Berkeley team behind JITEngine synthesizes custom inference engines for a specific Mac (chip, memory) and target model, arguing general engines that support any model on any hardware carry substantial overhead.
- Built with SkyDiscover; specialized engines made for a 24 GB M2 and a 48 GB M4 Pro, benchmarked on both machines
- Qwen3.8-Flash-Next runs 2.72x faster than slotstream on the 48 GB M4 Pro; Qwen3.6 beats llama.cpp with MTP by 1.36x on the 24 GB M2; 1.22x over rec at 252K-token context
- Users pick their Mac, memory budget and model, then get a one-command setup
More from Infra
- Voters rejected data centers in 40 of 41 local ballot measures — zacharynado · 2026-11-05
- Corebrum OS adds built-in VNC screen share, demoed remote-viewing an NVIDIA Thor robot from a Mac — chrismatthieu · 2026-11-05
- Modal partners with SGLang to serve Kimi K3.1 Pro on Day 0 — ying11231 · 2026-11-05
- Malaysia's deep AI hardware supply chain: Intel, AMD, Micron and more — davidyin44 · 2026-11-05
- Alibaba Qwen team's SpecOPD parallelizes on-policy distillation for faster speculative decoding — Qwen · 2026-11-05
- Revolut cut AI coding inference costs ~90% by self-hosting GLM on 56 B200s instead of Claude — sebkrier · 2026-11-05