Reddit bench: EPYC + multi-3090 rig runs Qwen Q8 at 94 tok/s with near-linear scaling to 8 streams
sleight42 · reddit · 2026-11-01
A user rented instances on vast.ai to benchmark local LLM serving on workstation-class hardware.
- Rigs: b7642 with 2× RTX 3090, dual AMD EPYC 7642 (96c/192t), 504GB RAM, Gen4 x8, $2.32/hr; g3090 with 4× 3090, EPYC 7K62, 251GB RAM, Gen4 x16, $0.79/hr.
- Key result: Strata running Qwen 3.8 Flash Next Unsloth Q8 hit 94 tok/s single-stream (121 tok/s tuned) and scaled nearly linearly to at least 8 concurrent streams (456 tok/s aggregate).
- Maya engine (GLM 5.3F / DeepSeek V4.1F): concurrency beyond C=1 is pointless — total output stays roughly fixed and extra streams just split it.
- The author distinguishes "sum of per-stream tok/s" from wall-clock measured aggregate, which diverges due to queueing.
Takeaway: big DRAM and decent CPUs beat fighting inflated VRAM prices (3090s selling above their 6-7 year old MSRP); vast.ai makes it cheap to test a rig before buying.
More from Infra
- Prediction: Local Models Will Be the Biggest AI Story of 2027, Disrupting Inference Business — ZeYanjie · 2026-11-01
- RAM up 7x to $429 in 18 months as Sam Altman says the invisible hand is failing — aakashgupta · 2026-11-01
- Blogger estimates xAI on track for $100B ARR by end-2026 as datacenter builds move up — mitchdeg · 2026-11-01
- Depth-growing 3B models beat dense training on GSM8K (19% vs 12%) while saving 26% pretraining compute — burny_tech · 2026-11-01
- From-scratch CUDA engine runs full GLM-5.3 on 3-6 DGX Sparks; steering vectors flip output language — burny_tech · 2026-11-01
- How NVIDIA's instruction scheduling model works: static latencies and scoreboards — mike64_t · 2026-11-01