Mismatched Tesla V100 pair (16GB+32GB) hits 1,377 prompt tok/s on Qwen3.8 27B locally
OkBase5453 · reddit · 2026-09-20
A Reddit user built a local inference lab from a mismatched pair of Tesla V100-PCIEs (16GB + 32GB, 48GB total) under Proxmox/LXC, benchmarking Qwen3.8 27B Q6KM at 1,376.9 prompt tok/s (2k context), 39.9 decode tok/s, and 1,221.5 pp tok/s at 16k.
Key tuning findings:
- Tensor split beats layer split: switching to tensor split with the 32GB card as main GPU lifted throughput from 981→1,377 pp tok/s and 23→40 tg tok/s; no split was worst.
- Q8 at 64k context with a 1:1.5 split still hits 664 pp tok/s.
- ikllama.cpp ran a 177B/6B-active MoE (Flash Next Q4, 104GB GGUF) across both cards at 26 tok/s decode.
- NInfer with NVFP4 27B reached 927 pp tok/s at 8k.
The daily-driver recipe: mainline llama.cpp + Qwen3.8 27B Q6 + tensor split + Flash Attention + Q8 KV + NUMA distribute + large batches — a case that old V100s remain surprisingly productive in today's GPU market.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04