Bonsai 2 9B hits 468% speedup on Mac, closing in on 1000 tok/s decode
HankYeomans · x · 2026-10-21
With heavy tuning of a custom dflash2 runner, TheDavidTai's team pushed Bonsai 2 9B to nearly 1000 tok/s on the mlx.fast leaderboard — the current record is 956.6 tok/s decode and 6818 tok/s prefill, 468% faster than the reference on Mac.
Key insight: results suggest a family of reduced-vocabulary, topic-specific speculative drafters could work if the prompt can be classified correctly. The leaderboard now has 120 promoted submissions from 13 solvers, with top runs largely produced with help from models like Opus 5.5, DeepSeek-V4 and GPT-6 Astra.
More from Infra
- CoreWeave launches Physical AI Launchables: run robotics and simulation workloads with one command — ayushthakur0 · 2026-10-21
- NVIDIA locks up massive ABF substrate capacity again, cementing its win in the AI compute cycle — BenBajarin · 2026-10-21
- php2shell Flaws Hit ~70% of Websites; Cloudflare Shipped WAF Rules 9 Hours Early — Cloudflare Blog · 2026-10-21
- SemiAnalysis: 1,730 solar pullers yield ~30 GWp, only 4-7 GW of firm datacenter power — BenBajarin · 2026-10-21
- Frozen 16 MB vocabulary slashes small LLM size to 46 MB with no quality loss — pmttyji · 2026-10-21
- llampart v2: open-source local LLM chat UI with charts, Mermaid and MCP support — mossy_troll_84 · 2026-10-21