Latent Sparse Attention on Apple Silicon: 31% faster TTFT at 128k with a hybrid prefill/decode split
GabGarrett · x · 2026-11-05
The rapidmlx team got Latent Sparse Attention (LSA) running end-to-end on Apple Silicon and found a hybrid recipe (Qwen3.8-27B on M3 Ultra, 128k context):
- Time to first token: 640s → 440s (-31%)
- Peak memory: 40.7 GB → 28.8 GB (-29%)
- MTP decode speed: zero regression
Key insight: pure LSA matches the paper's prefill scaling (1.16× at 64k, 1.41× at 128k) but slows generation — per-token routing overhead hurts when decode is memory-bandwidth bound. The fix: sparse attention for compute-bound prefill, dense MTP for bandwidth-bound decode.
LSA itself is an open release from jcrwhittington's team: at 10M tokens, finding the right tokens takes 98% of attention-layer time; LSA makes sparse attention up to 22.7× faster and full attention 72.4× faster while matching full-attention accuracy at ultra-long context. Qwen3.8-27B + GLM-5.3-Flash benchmarks across the M-series lineup are next.
More from Infra
- Google's 1999 server rack: wooden shelves, tangled cables, and a lesson in just building — Yamapama · 2026-11-05
- Olki ditches LLM-based agent monitoring for strict type systems and effects algebra — bigdata · 2026-11-05
- tiny-llm: Open Course Builds a Mini vLLM Inference Engine on Apple Silicon — alexcovo_eth · 2026-11-05
- DeepSeek charges double for peak-hour API calls, and off-peak scheduling halves your token bill — AccBalanced · 2026-11-05
- NVIDIA RTX PRO 5500 Blackwell lists at $17,999, pricier than RTX PRO 6000's $16,000 — panchovix · 2026-11-05
- Nemotron 3.5 Bolt E4B hits 27.7 tok/s on Jetson Nano, 1.8x faster than llama.cpp — Course_Latter · 2026-11-05