Running Qwen 27B via llama.cpp RPC on old GPUs: benchmarks and tuning tips
BigPoppaK78 · reddit · 2026-08-18
The author shares benchmarks and optimization tips for running Qwen 3.8 27B via llama.cpp RPC across a 5070 Ti and 1080 Ti over Gigabit Ethernet.
- Performance: Achieved 36 t/s generation speed or 560 t/s prefill speed at 12k context depending on configuration.
- MTP Tuning: Enabling MTP hits prefill speed by 30%; place the strongest GPU at the end of the RPC chain to handle MTP steps.
- KV Quantization: Enabling KV quant on the draft model reduces available context, contrary to expectations.
- Network & Batching: Enable jumbo frames; optimal batch sizes vary (512/64 for prefill, 1536/256 with MTP).
- Hardware: An 8GB VRAM card with sufficient bandwidth works well as a secondary card.
More from Infra
- DeepSeek V4 Flash Benchmarks: n_max=3 Yields 1.39× Speedup — Responsible_Pain3278 · 2026-08-18
- NVIDIA pledges up to $105B in project guarantees, raising questions about artificial growth fueled by self-funding chip sales — heypearlai · 2026-08-18
- Inference > Training: LLMs are served year-round but trained once or twice — prajdabre · 2026-08-18
- Testing MTP combined with ngram-mod for coding speed — YetAnotherAnonymoose · 2026-08-18
- Tesla's Optimus: Leveraging FSD and Auto Supply Chain for Humanoid Robots — 创业邦 · 2026-08-18
- With USB4STREAM merged into Linux 7.2, are inference runtimes adopting it? — voyager256 · 2026-08-18