Local Multi-GPU Cluster Concurrently Runs Multiple Open-Source LLMs with Performance Stats
Any-Lingonberry7411 · reddit · 2026-08-07
A developer showcased a high-end local multi-GPU cluster used for agentic coding, sharing performance metrics for concurrently running multiple open-source LLMs.
- Hardware: The main rig features an RTX 6000 Pro Blackwell (96GB), 2x RTX 5090, 1x RTX 4090, and 3x AMD R9700, alongside 96GB RAM. A Strix Halo laptop and an RTX 3090 rig act as secondary machines.
- Concurrency: The setup can simultaneously run models like DeepSeek-V4-Flash (45 t/s), GLM-4.7-flash (20 t/s), and Gemma-4-12B (25 t/s). Pooling all resources allows running massive models like GLM-5.2 or Kimi-K2.7-Code.
- Bottleneck: The author notes that seamlessly switching between a "fast worker" and a "deep thinker" is currently difficult, as evicting and loading models kills the cache and slows down the workflow. They are seeking upgrade advice.
More from coding & agent
- Opus 5.5 + Three.js demo stuns with water physics, fish and blossoms — nikola_mr64990 · 2026-10-16
- Grok-built linear regression bot claims $508K PnL in 125 days on Polymarket — Roger_M_Taylor · 2026-10-16
- Karpathy's knowledge engine workflow: Claude Code turns raw notes into a self-updating wiki — nikola_mr64990 · 2026-10-16
- Sierra and Meta unveil Personal Agent Protocol with Zapier as launch partner — sujingshen · 2026-10-16
- ArcKit adds 102 AI-narrated short films for commands across 14 community overlays — mcraddock · 2026-10-16
- Personal agents live or die on chaining tasks, not single-task ability — sujingshen · 2026-10-16