PCIe x1 pushed to the limit: concurrently serving 27B, 9B, 2B vision LLMs on Frankenstein hardware
FieldMouse-AI · reddit · 2026-10-24
Reddit user lvgale shares concurrency benchmarks from their cobbled-together standby node, serving three vision LLMs of different sizes simultaneously, with a photo of the full build.
- Hardware: i7-6700K + four mismatched GPUs—RX 580 8GB (x16), RX 570 8GB (x1), GTX 1660 Super 6GB (x2), GTX 1070 8GB (x1); the bottleneck is activations crossing x1/x2 lanes between entirely different AMD and NVIDIA architectures
- Software: Slipstream, a custom Nuitka-compiled Python orchestration engine, using a mixed-vendor Vulkan backend with Flash Attention for stability across the fragmented topology
- Concurrent throughput: 27B vision model at 3.82 t/s, 9B at 4.75 t/s, 2B at 12.47 t/s, seamlessly handling OCR and complex vision parsing throughout
The author says keeping the 27B model stable on this mixed-architecture, low-bandwidth topology without crashing the Vulkan driver is itself a major win—useful reference for low-cost local inference setups.
More from Infra
- Top law firm reportedly ditches Harvey and cloud APIs, self-hosts Kimi and DeepSeek — SumitGup · 2026-10-24
- Tesla Energy Q3: deployments up 10% but revenue down 16% as prices collapse — pdamodaran · 2026-10-24
- Mandiant: Some PHP Vulnerability PoCs Achieve RCE on mod_php, Patch Now — cyb3rops · 2026-10-24
- humans& W4A16 kernel hits 98% Tensor Core utilization, 10% faster than FlashInfer NVFP4 — niloofar_mire · 2026-10-24
- Chanos questions AI compute boom as Blackstone shops data centers while neocloud rates jump 50% — GaryMarcus · 2026-10-24
- Cancer center's $300-400K GPU dilemma: RTX Pro 5000 pairs vs one 8-GPU box for AlphaFold — starkruzr · 2026-10-24