Pooling spare RAM across old devices to run 30B local models as memory prices soar
Medicine_Blogscanner · reddit · 2026-09-25
With RAM prices climbing, a Reddit user pools spare memory across old devices instead of buying a big machine.
Test: a 30B model split between a mini PC (12GB) and Mac mini (5.5GB) hits mid-30s tokens/sec at 1s latency over plain LAN. They also tested RAG: the model hallucinated about a newly approved rare-disease drug, but after feeding a research PDF — stored as shards in RAM across both devices, not on disk — it answered correctly.
Key finding: the inference machine and knowledge-base machine don't need to be the same; a weaker box can just hold document shards, with a recovery step that pulls backups and re-splits if a shard device goes offline. The author asks whether all-RAM RAG indexes hold up at larger scale vs on-disk vector DBs.
More from Infra
- Deep Inference-Query Engine Integration: Custom Scheduler and Workload-Aware KV Cache for Prefill-Only AI Filters — charles_irl · 2026-09-25
- Diffusion LLM goes production: Augment Code's Mercury 2.5 switch cuts latency 82%, cost 90% — cen6wkf · 2026-09-25
- Oracle says force majeure notice doesn't signal data center delays, rent still due — AIFlow_ML · 2026-09-25
- SemiAnalysis turns more bullish on China's WFE localization after CSEAC 2026 — zephyr_z9 · 2026-09-25
- Meta Muse runs on just 2 cores of an AMD EPYC server CPU — firstadopter · 2026-09-25
- M5 Ultra 80-core tested with GLM-5.3-Flash: RAM is great, GPU is the bottleneck — dreamingwell · 2026-09-25