Mismatched Tesla V100 pair (16GB+32GB) hits 1,377 prompt tok/s on Qwen3.8 27B locally
OkBase5453 · reddit · 2026-09-20
A Reddit user built a local inference lab from a mismatched pair of Tesla V100-PCIEs (16GB + 32GB, 48GB total) under Proxmox/LXC, benchmarking Qwen3.8 27B Q6KM at 1,376.9 prompt tok/s (2k context), 39.9 decode tok/s, and 1,221.5 pp tok/s at 16k.
Key tuning findings:
- Tensor split beats layer split: switching to tensor split with the 32GB card as main GPU lifted throughput from 981→1,377 pp tok/s and 23→40 tg tok/s; no split was worst.
- Q8 at 64k context with a 1:1.5 split still hits 664 pp tok/s.
- ikllama.cpp ran a 177B/6B-active MoE (Flash Next Q4, 104GB GGUF) across both cards at 26 tok/s decode.
- NInfer with NVFP4 27B reached 927 pp tok/s at 8k.
The daily-driver recipe: mainline llama.cpp + Qwen3.8 27B Q6 + tensor split + Flash Attention + Q8 KV + NUMA distribute + large batches — a case that old V100s remain surprisingly productive in today's GPU market.
More from Infra
- Jev-style parallel structured inference makes 350M model 63× faster, code released — helloiamleonie · 2026-09-20
- Dev builds SLO-aware inference router with Jev to pick the optimal LLM per request — ai · 2026-09-20
- Memristive Networks Learn by Reorganizing Themselves: When Material Is the Model — bravo_abad · 2026-09-20
- Crusoe signs multiyear cloud deal to run dedicated Nvidia GB300 clusters for Perplexity — Beth_Kindig · 2026-09-20
- Entire arXiv uploaded to Hugging Face: 3.15M papers, every version, 16TB in LaTeX/PDF/HTML — secemp9 · 2026-09-20
- SF Compute interviews exchange legend Rich Jaycobs on how to build a compute futures market — andriy_mulyar · 2026-09-20