Running a Q2-quantized LLM on a 24GB MacBook Air: 49 t/s via SSD expert streaming

Inside-Pen-8395 · reddit · 2026-10-25

The author ran an ISTA GSQ-RCO Q20 quant of Qwen3.8-Flash-Next on a 24GB MacBook Air M5 using Splash 1.4.0 plus a personal fork (800 lines, all behind env flags), with Flash-Next-DFlash2 speculative decoding (draft acceptance: 70% code, 80% tool calls) and 256K context.

Key trick — expert streaming: only 6,800 of 24,576 expert slots (28%) stay resident in 8.8 GB; the rest are read from SSD on demand.

Performance:

Stock Splash: 28.8 t/s average. Adding cache-aware routing (protect top-4 experts, bias picks 5–10 toward resident ones) → 43.1 t/s. Adding desk-first overlap (GPU computes resident experts while missing ones stream in; output token-for-token identical) → 48.8 t/s, 70% over stock.
Real agent use: 45 t/s with zero prompt-cache losses (cache persists on disk; only new tokens read per turn).
Cold prefill: 32K tokens in 62.4s (522 t/s); 63K in 130.4s (485 t/s).

Minimal quality cost: 17/20 vs 16/20 on LiveCodeBench (single run, ±1 is noise); agent and long-doc tests identical. A lazy-vision experiment loads the 864 MB vision tower only for image requests, saving 656 expert slots and 5.5% decode cost. The author notes it beats Tiel at a runnable quant and is much faster than a 27B for their use case.

Original post →

Running a Q2-quantized LLM on a 24GB MacBook Air: 49 t/s via SSD expert streaming

More from Infra