Dual A100 80GB llama.cpp setup: 50-70 tk/s solo, drops to 10-15 with 2-3 users
vitamins1000 · reddit · 2026-10-13
A Reddit user asks for help tuning a dual A100 80GB setup (Proxmox GPU passthrough on EPYC 7402P, PCIe Gen4 x8 due to odd DPU cards) running Qwen3.8 Flash UD-Q4KXL with a Q80 MTP draft head in llama.cpp, configured with 7 parallel slots x 262k context (1.8M total) and q80 K/V cache.
Real-world single-user throughput is 50-70 tk/s, but generation craters to 10-15 tk/s with 2-3 concurrent sessions. The use case is long-session agentic coding (running Linux commands, probing hardware). They prefer MoE models and are weighing a switch to vLLM or running separate qwen3.6-35b-a3b instances per card.
More from Infra
- Blog post shrinks from 158KB HTML to 17KB Markdown for AI agents — colinmcnamara · 2026-10-13
- Zuckerberg: an H100 is worth more than last year as AI compute stays scarce — inductionheads · 2026-10-13
- Arm: AI data centers need 3 CPU roles, market to grow from $52B to $300B by 2030 — firstadopter · 2026-10-13
- Every AI compute fix just moves the bottleneck: stacking, optics, big dies each spawn new constraints — BenBajarin · 2026-10-13
- Meta's Hyperion to be world's first 5GW datacenter, a $400B bet on 4,000 acres — bookwormengr · 2026-10-13
- Home AI lab with two DGX GB300s serves DeepSeek at 5,774 tok/s, all solar — julianharris · 2026-10-13