Streaming a 341GB model on a 128GB Mac: two macOS fixes made first token 8x faster
Chida82 · reddit · 2026-10-12
Developer Chida82 details two macOS-level fixes in sf-ds4-1flash, his fork of antirez's ds4 running DeepSeek V4.1 Flash on Metal, streaming a 341GB Q2 GGUF from SSD on a 128GB M5 Max.
1. FNOCACHE isn't fully "no cache" on APFS
- Reads not starting on a 16KB page boundary still get partially cached; expert weights sit 3MB apart, so they almost never align. Testing showed 982 of 3731 pages stayed cached, evicting decode weights.
- Fix: read the partial first page separately, then from the boundary.
- Result: first token after an 8K prompt 1.23s → 0.15s; +12.5% when appending 1.5K tokens.
2. Poll a mailbox instead of waiting on the command buffer
- Getting expert IDs via command-buffer completion costs 48µs; spinning on a shared buffer written by the router kernel takes 20µs, applied on every layer of every token. Side finding: GPU writes become CPU-visible when the command buffer ends, not the kernel.
- Result: decode +5.5% at 2K, +6.2% at 8K context.
Vs upstream ds4 (same GGUF, same flags): prefill +45% to +99%; decode 30 tok/s at 2K, 26 at 32K (+62% to +78%); first token 0.15-0.2s vs 2-3s. Logits are bit-checked after every change; output is token-identical.
Caveat: single machine; decode settles around 24 tok/s after an hour of heavy use. Repo: github.com/Chida82/sf-ds4-1flash
More from Infra
- WSJ: Inside Silicon Valley's compute wars — Google paid $2.7B for Shazeer, then lost him to OpenAI — dotey · 2026-10-12
- Qualcomm AI uses entropy and CKA to pick layers, beating EAGLE3 defaults in speculative decoding — Qualcomm-AI-Research · 2026-10-12
- Inference is a permanent reverse auction: endless supply and easy switching keep grinding prices down — joecole · 2026-10-12
- 'The most important chart for the AI economy': models commoditize, compute wins — FinanceYF5 · 2026-10-12
- Anthropic's Claude Managed Agents: Decoupling Brain from Hands Cut First-Token Time by 90% — AI Engineer · 2026-10-12
- Google's Token Volume Grew 7x to 3.2 Quadrillion a Month — How Cerebras and Gimlet Keep Inference Up — AI Engineer · 2026-10-12