Four 8GB 3060 Tis run Qwen3.8-27B at 120 t/s with 150k context via tensor parallel
DontWinFrensWthSalad · reddit · 2026-09-27
A Reddit user skipped buying a new GPU and instead built a 4x RTX 3060 Ti rig (8GB VRAM each, power-limited to 110W) from old mining cards, with surprisingly strong results.
Key approach
- The crucial piece is tensor parallelism: llama.cpp doesn't support it, so they used Turboderp's ExL3 first, then moved to HyperQwen + vLLM, which supports TP=4 on Ampere cards.
- ExL3 + MTP served Swift-Qwen3.8-27B at 70 t/s with 196k context.
- With vLLM running the W4A16 fast variant: 150k context at 120 t/s in bf16; KV quantization (kv8) unlocks the full 262k context but drops speed back to 70 t/s.
- Concurrency costs almost nothing: one blazing-fast agent or two concurrent agents sharing a big context window.
The takeaway: instead of spending $1000+ on a single 3090, upgrading motherboard/CPU and reusing old cards with multi-GPU tensor parallel is a viable local inference path.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11