8x P100 llama.cpp: Qwen3.8 Flash Q4 Throughput Swings Wildly With Slot Count
ILoveDangerousStuff2 · reddit · 2026-10-28
The author runs Unsloth's Qwen 3.8 Flash Next Q4 on a G292-Z20 with 8 P100 PCIe GPUs, Manjaro on bare metal and a self-compiled llama.cpp, aiming for good concurrency for agentic use.
- Single-stream generation is 20–24 tps with prefill around 400–500 tps; enabling MTP pushes single-stream to 33 tps.
- With 16 slots, total throughput peaks near 80 tps at 11 concurrent requests, then drops sharply.
- Changing the slot count always produces some concurrency level where throughput collapses, recovers, then collapses again.
- Best result: 20 slots at 13 concurrency, 91 tps total, but only 30 tps just before and after.
- The pattern is stable for a given slot count but varies unpredictably across slot counts.
The author suspects there is still optimization headroom and asks the community what could cause this.
More from Infra
- Will local-model progress outrun GPU depreciation? Reddit debates the buy-now window — Sensualities · 2026-10-28
- Dev cuts local AI inference power from 100W to 25W with only a 1.9s→2.8s slowdown — AryanEmbered · 2026-10-28
- SK Hynix posts $62.8B quarterly operating profit at 80% margin as AI memory demand booms — mhdfaran · 2026-10-28
- SK hynix plans ~KRW 100T shareholder returns in FY2026 on AI-driven memory boom — firstadopter · 2026-10-28
- SK hynix: Global CSP CapEx to grow 60%+ YoY, AI DC infra market at 40% CAGR through 2030 — firstadopter · 2026-10-28
- OpenAI plans to use Google's TPUs, The Information reports — zephyr_z9 · 2026-10-28