DeepSeek V4.1 streaming EXL3 packs bring ~70 tok/s local inference to 128GB Macs

HankYeomans · x · 2026-11-05

Developer TheDavidTai released updated turboexl3 + mlx-serve streaming packs for DeepSeek V4.1-Flash-1017, targeting M5 96GB/256GB devices:

Performance: 70 tok/s decode, up to 1900 tok/s prefill, around NVFP4-equivalent quality, usable as an everyday local agent
Consistency: speed stays within 10% of the rated rate across the full 1K-256K context range
Approach: trunk stays in RAM while routed experts and Engram tables stream from SSD on demand, so even a 128GB Mac can serve the 408GB 2.81bpw pack; quants use turboderp's self-corrected pipeline distilled from lossless DS4.1, beating vanilla EXL3 and matching Mia-AI Lab's EAQ Spark
Benchmark charts: M5 Max 128GB results across 1.68-2.81bpw packs — prefill/decode throughput, time-to-first-token, and KLD accuracy (2.81bpw mean KLD 0.0042 in the ≥0.95 confidence bucket, slightly better than EAQ v2.9)

Weights are live on Hugging Face under MIT; packs will ship in the next mlx-serve release.

More from Infra