Dual 3090 Qwen Deployment: Switching Split Mode Quadruples Prompt Processing Speed
DjCanalex · reddit · 2026-08-07
A developer shared an in-depth performance tuning guide for deploying a Qwen 27B model using llama.cpp on dual RTX 3090 GPUs.
Issue Isolation
Previously, the author used --split-mode tensor, which yielded decent token generation (tg) speeds. However, he discovered that prompt processing (pp) was falling back to the CPU, bottlenecking the pp speed at around 400 t/s.
Performance Leap
Benchmarking revealed the hardware was capable of 1600+ t/s. By switching to --split-mode layer, the GPU took over sampling. While token generation slightly dropped from 60-70 t/s to 40-55 t/s, prompt processing skyrocketed by more than 4x, drastically improving overall long-context inference efficiency.
More from Infra
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16