Local Qwen3 8B Flash on RTX 6000 Pro: 2000 tps prefill but only 40 tps decode
AppealSame4367 · reddit · 2026-09-04
A user runs Qwen3 8B Flash on a rented RTX 6000 Pro via ikllama (3 slots, 200k context, IQ4 quant, q8 KV cache): 2000 tps prefill but only 40 tps decode for a single request, dropping to 10-20 tps decode with 2-4 parallel requests. vLLM recipes didn't help; they're asking for better configs for 2-4 slots at q4+ quants.
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20