DeepSeek-V4 in 64GB RAM decodes at just 9-15 tok/s — a capacity trick, not an inference win
petrusenko_max · x · 2026-11-09
Analyzing the llama-modchip repo's claim of fitting 56.56 GiB of DeepSeek-V4 into 64GB of RAM: it's a 1.71 bpw extreme-quantization capacity trick, not an inference win. Decode speeds of 9.1–14.7 tok/s can't carry concurrent agent traffic, while the repo's own GPU-offload path hits 80.80 tok/s. Don't judge on-device deployments by headline numbers alone.
More from Infra
- AbridgeKV: latent KV cache compression matches full accuracy at 1/4 cache, 3.2× faster — DimitrisPapail · 2026-11-09
- Developer builds Shura: a local coding assistant on one RX 7800 XT hitting 78 tok/s that should improve with use — HIKKIAN · 2026-11-09
- Altman says home DRAM is a waste as memory prices soar; critics blame OpenAI's datacenter buildout — jonerp · 2026-11-09
- Public GitHub pushes double in one year to 408M per quarter amid AI-generated code surge — DimitrisPapail · 2026-11-09
- McKinsey: multi-step agents burn 5-30x more tokens than chat — ingliguori · 2026-11-09
- Unitree G1 + Mac mini demo reportedly hides 200 H100s behind it — yongqianme · 2026-11-09