RAM-Pooling Across 3 Old Devices Runs a 40B Model at 16 tok/s — With a Counterintuitive CPU Finding
Medicine_Blogscanner · reddit · 2026-09-16
The author used their open-source ramdeck-core to pool RAM across three household devices — a mini PC with an RTX 3060, a 16GB Mac Mini, and an old laptop — and successfully loaded a 25GB 40B model no single machine could hold.
The sharding logic maxed the 3060 first, spilled into the Mac's unified memory, then to CPU. Counterintuitively, routing CPU spillover to the old laptop ran faster than keeping it on the mini PC's own CPU — possibly because sharing GPU+CPU inference on one node hurts GPU utilization, so offloading to a separate slower box wins entirely.
Total load took 2.5 minutes; inference hit 16 tok/s with 3.6s latency — genuinely usable for chat. Repo open-sourced with a video demo.
More from Infra
- Brad Gerstner: AI buildout hinges on revenue keeping its steep growth curve — markjeffrey · 2026-09-16
- Reported: NVIDIA hardware delivers 40% more throughput at the same power — karlfreund · 2026-09-16
- NVIDIA Shares New Vera Platform Data at AI Infra Summit — karlfreund · 2026-09-16
- Token usage forecast to hit 4 quintillion by 2030, agentic AI to drive most of it: Evercore — Beth_Kindig · 2026-09-16
- NVIDIA's Ian Buck shares Groq inference performance data at AIInfraSummit — karlfreund · 2026-09-16
- MLPerf Inference v6.1 results imminent: 30 submitters, new accelerators and platforms — TheKanter · 2026-09-16