Can llama.cpp RPC span a Mac mini, Mac Studio and dual RTX 3090s to run a 120B model?
ruffus_or · reddit · 2026-11-22
A user is planning a three-machine distributed AI cluster to run a 120B+ parameter model: a Mac mini M4 Pro (64GB unified memory), a Mac Studio M4 Max (36GB unified memory) they're considering buying, and a Windows PC with 2× RTX 3090 (48GB VRAM) plus 64GB DDR4 — 212GB of physical memory in total, though not a single shared pool.
They intend to combine Metal, CUDA and CPU offload via llama.cpp RPC or similar, with Thunderbolt 5 between the Macs and 10Gb Ethernet to the Windows box, and are asking whether anyone has pulled off something comparable.
More from Infra
- An AI bubble fable: from free cash flow to private credit to annuity blowup — Svwang1 · 2026-11-22
- One RX 7900 XTX runs 524k context at 767 tok/s, 3x faster than llama.cpp — W61k3r · 2026-11-22
- How GPUs organize work: kernel launches, thread blocks and scheduling explained — blaizedsouza · 2026-11-22
- Alibaba's T-Head files for HK IPO at up to $80B, rivaling Baidu's Kunlunxin — 创业邦 · 2026-11-22
- OpenAI's GPT rewrites AMD flagship GPU RTL, gaining up to 14% PPA in 5 months — 新智元 · 2026-11-22
- LBNL report: data center load growth did not drive up residential electricity prices (2019-2025) — Eliv_nurotic · 2026-11-22