Running DeepSeek-V4-Flash on 64GB Mac: Small cache beats large

cowboy-bebob · reddit · 2026-08-22

Ran DeepSeek-V4-Flash (284B/165GB) on an M5 Pro Mac with 64GB RAM for two weeks, streaming experts from SSD.

Key Findings:

Streaming loses no quality: Full-corpus wikitext perplexity is 6.1250 vs. published 6.1262.
Small cache beats large: 8GB cache achieved 2.04 tok/s vs. 1.23 tok/s for 32GB. The OS page cache does the real work. Replicated in ds4 engine: 8GB beat 32GB on decode, prefill, and RSS, hitting 11.4 tok/s in 5.1GB resident memory.
Speculative decoding & prefetch hurt throughput: In the IO-bound regime, extra bytes cost more than latency saved. Prompt-lookup speculation measured 0.44–0.96x. Pre-gated prefetch (70-77% accuracy) did not increase speed.
2-bit quantization failed: Codebook 2-bit (AQLM class) failed quality gate (+9.4% perplexity), equal to plain affine 2-bit.

More from Infra