Running 748B DeepSeek-V4.1-Flash on a 24GB Mac: streaming experts off SSD hits 1.36 tok/s
solyarisoftware · x · 2026-09-15
Developer mfranzon open-sourced a full project running DeepSeek-V4.1-Flash on a 24GB MacBook (Apple M5):
- The model totals 748B parameters (552B MoE backbone + 196B engram n-gram tables), about 426GB — yet fits in 14GB of RAM by streaming experts off the SSD on demand instead of holding them in memory.
- Inference speed was optimized from 0.74 to 1.36 tok/s — far from practical, but the author shares the whole journey.
- The repo includes code, acceleration notes (ACCELERATION.md), architecture docs, streaming findings, and phase-by-phase writeups; 310GB free SSD is needed (shards are downloaded and deleted as you go), Python 3.12, no full Xcode required.
A rare complete reference for local deployment on resource-constrained hardware.
Related event: Devs Squeeze Giant DeepSeek Model onto Consumer Hardware(2 posts)→
More from Infra
- Researchers encode 200+ properties of SGLang inference server, uncover 10 bugs — LingmingZhang · 2026-10-31
- Classical infrastructure gets more important, not less, in the age of AI agents — graceisford · 2026-10-31
- Lux Capital hosts closed-door dinner on AI infra, agents and security with Cognition and Modal — graceisford · 2026-10-31
- FastVideo ships install guide for NVIDIA DGX Spark, no aarch64 wheels needed — victormustar · 2026-10-31
- Intel shifts 18A capacity to Intel 3 to expand Granite Rapids as Clearwater Forest ships — zephyr_z9 · 2026-10-31
- Modal launches Sticky Sessions: serverless containers that keep state across requests — graceisford · 2026-10-31