Dev runs 284B-parameter DeepSeek-V4-Flash on CPU only, squeezed into 64GB DDR5 RAM
solyarisoftware · x · 2026-11-07
Developer synchopate released llama-modchip, a modified llama.cpp inference stack that runs DeepSeek-V4-Flash (284.33B total / 13B active) entirely in system memory with zero GPUs.
- Hardware: CPU + 64GB DDR5 only, 256K context configured
- Custom AVX2/AVX-512 CPU kernels
- Optimized quantization cuts model storage 26.4% (76.87 → 56.56 GiB), fitting the 64GB limit
- CPU decode at 9.1–14.7 tok/s, prompt processing 44 tok/s — reported as usable
- Open source on GitHub
A notable proof that large MoE models can run locally on CPU-only machines.
More from Infra
- NVIDIA's first co-packaged optics Ethernet switch, Spectrum-X Photonics, enters full production — TinfoilTricorn · 2026-11-07
- This week's open training stack: MoE training handbook, harness adapters, and a harness leaderboard — ariG23498 · 2026-11-07
- Musk: space will be the cheapest place to run superintelligence within 2-3 years — TinfoilTricorn · 2026-11-07
- JITEngine synthesizes custom LLM inference engines for Apple Silicon, claims up to 2.7x speedup — AccBalanced · 2026-11-07
- Dev predicts 2027's local inference boom: integrated hardware + OS + inference stack, not just GPUs — vincent_koc · 2026-11-07
- vLLM Office Hours Hits 59 Sessions: 76 Speakers from 24 Organizations and Counting — vllm_project · 2026-11-07