Running Qwen 27B via llama.cpp RPC on old GPUs: benchmarks and tuning tips
BigPoppaK78 · reddit · 2026-08-18
The author shares benchmarks and optimization tips for running Qwen 3.8 27B via llama.cpp RPC across a 5070 Ti and 1080 Ti over Gigabit Ethernet.
- Performance: Achieved 36 t/s generation speed or 560 t/s prefill speed at 12k context depending on configuration.
- MTP Tuning: Enabling MTP hits prefill speed by 30%; place the strongest GPU at the end of the RPC chain to handle MTP steps.
- KV Quantization: Enabling KV quant on the draft model reduces available context, contrary to expectations.
- Network & Batching: Enable jumbo frames; optimal batch sizes vary (512/64 for prefill, 1536/256 with MTP).
- Hardware: An 8GB VRAM card with sufficient bandwidth works well as a secondary card.
More from Infra
- Phison CEO calls CXMT memory scare a 'ghost story', warns of materials shortage — BenBajarin · 2026-10-16
- Ben Bajarin: The Future of Compute Is Fungible as Hyperscalers Design for Workload Flexibility — BenBajarin · 2026-10-16
- AirLLM runs frontier-size models layer-by-layer: 671B on 12GB VRAM — Aiden_Tech_Ai · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Q4-quantized coder model on 8-core CPU builds interactive solar system demo in one shot — IntrepidMindExplorer · 2026-10-16
- TokenSpeed MLA 2 ships for Vera Rubin: up to 3.44× faster Kimi K3 decode — zhyncs42 · 2026-10-16