Running Qwen 27B Locally on 2× RTX 5070 Ti: A Cost-Effective Inference Setup
val_in_tech · reddit · 2026-08-06
The author shares a cost-effective local LLM inference setup, running the Qwen 3.6 27B model using two RTX 5070 Ti GPUs. A single 5070 Ti provides 896 GB/s of memory bandwidth, making it highly suitable for bandwidth-bound dense models.
Performance metrics:
- Pure VRAM Inference: With VLLM TP2, CUDA graphs, and MTP, it handles 52k context, 4.5k prefill, and achieves 95 tps generation speed.
- KV Offload Mode: By offloading KV into RAM, the context length extends to 163k (a 3x increase) with only a 5-10% drop in generation speed (85-90 tps).
The author suggests that for users accustomed to RTX 3090-level performance, the RTX 5070 Ti is a powerful option that supports all modern features at a relatively affordable price point.
More from Infra
- Run 193B parameter model on 24GB RAM: 8 specialist models with dynamic routing — Similar_Wealth_1850 · 2026-08-06
- Truespar Launches Paddock: High-Performance Local LLM Engine for NVIDIA GPUs — wandedob · 2026-08-06
- Breaking the AI Memory Wall: CXL Moves Toward Commercial Deployment — BenBajarin · 2026-08-06
- Redefining Productivity: 'Intelligence Per Watt' (IPW) as the New Economic Metric — NinaDSchick · 2026-08-06
- Rack-Scale AI Infrastructure Accounts for Only ~10% of Installed Base — BenBajarin · 2026-08-06
- Brevis Treats Lossless Tensor Compression as Program Synthesis, Cutting Storage by 33% — SingaporeManagementUniversity · 2026-08-06