85GB DeepSeek-V4-Flash Runs at 3 tok/s on a 12GB RTX 3060 via Disk-Tier Overspill
Chekhovs_Shotgun · reddit · 2026-09-27
The author built Overspill, a disk tier for FreeToken, to run MoE models that don't fit in RAM, inspired by Colibri's expert placement across disk/RAM/VRAM. Tested on an RTX 3060 12GB, Ryzen 9 7900, 64GB DDR5-6000 and NVMe under Windows 11 + WSL2, with DeepSeek-V4-Flash REAP-150B (85GB, FP4 experts).
Measured results (same machine, cold start, greedy decoding)
- Decode: Overspill 2.8–3.4 tok/s vs llama.cpp 0.4–0.5 tok/s vs Colibri 1.1–1.2 tok/s
- TTFT on a 6.4k prompt: 102s / 373s / 26 min
- First short prompt after cold start: 43s / 44s / 25s; next short prompt: 10s / 40s / 18s
Key changes
- Memory-mapping experts that don't fit in RAM so the OS page cache acts as another tier
- madvise(WILLNEED) to read each layer's routed experts in parallel with large reads instead of page-fault-by-page-fault
- Keeping embedding/output layers in RAM to free VRAM
- Larger prompt chunks to reduce expert streaming
- Running short prompts on CPU instead of moving the full expert set through the GPU
- Disk expert loading got roughly 4x faster; also fixed a FreeToken checkpoint converter OOM on models larger than RAM
Sanity checks and failures
- Byte-identical output to stock FreeToken on Qwen3.6-35B-A3B (which fits in RAM)
- The DeepSeek model passed a coding test and multi-turn tool-calling, though no comprehensive eval was done
- Prefetching next-layer experts from RAM to GPU was 9–29% slower on the 3060, possibly due to transfer/compute contention
The author stresses this is a single-machine, single-request proof of concept; the disk path runs expert math on CPU (AVX-512), so speed depends heavily on CPU, RAM bandwidth and SSD, not just the GPU.
More from Infra
- RAM prices up 500%: buy local AI hardware now, says analyst roundup — julianharris · 2026-09-27
- UK grid power shortfall could delay a major AI project by years — rvp · 2026-09-27
- FT: OpenAI projects negative free cash flow of $278B through 2030 on compute spending — Beth_Kindig · 2026-09-27
- IIT Delhi says it has built India's first indigenously designed micro-GPU — rvp · 2026-09-27
- Musk: China Will Solve Its Compute, Lithography and Chipmaking Constraints in 2-3 Years — haider1 · 2026-09-27
- Dev argues Vercel is 'unjustifiable' now that agents can safely drive Cloudflare — generativist · 2026-09-27