Running an 816B-parameter model on two Mac Studios: $40k instead of $500k of compute

charles_irl · x · 2026-10-16

ekzhang1 showed how to run a trillion-parameter-class model—normally needing 4xB300 ($500k of datacenter compute)—on roughly $40k of consumer hardware.

• Model: 816B params (42B active), quantized to nvfp4 (493 GB of weights).
• Hardware: two Mac Studios with 512 GB unified RAM each, weight/attention sharding over both machines, TP=2, connected via Thunderbolt RDMA.
• Performance: single-request concurrency, memory-bandwidth bound (1.2 TB/s per GPU vs 8 TB/s on Blackwell).
• He admits it's not practical or cost-effective at unit prices (tiny batch size), but "it does run at all, which is pretty neat." Context: Meta released Muse Spark Open weights, optimized for new Mac Studios in collaboration with Apple.

Original post →

More from Infra

• "A datacenter is a bank": CoreWeave drama revives calls for算力 supervision — arthurcolle · 2026-10-16
• DOT-MoE: converting dense models into sparse MoE via optimal transport — burkov · 2026-10-16
• Stanford CS336: A 7B model needs 14GB for weights but 84GB to train — stanfordnlp · 2026-10-16
• New NVFP4 trick: compressing block scales saves 45-48% per expert matrix, bit-exact — TheZachMueller · 2026-10-16
• AWS extends Claude Sonnet 3/3.5/3.6 EOL on Bedrock to November 13, 2026 — repligate · 2026-10-16
• After Apple and Meta CEOs tout Mac Silicon inference, a $10B distributed network idea resurfaces — gajesh · 2026-10-16

Infra channel →