Dual 3090 Qwen Deployment: Switching Split Mode Quadruples Prompt Processing Speed
DjCanalex · reddit · 2026-08-07
A developer shared an in-depth performance tuning guide for deploying a Qwen 27B model using llama.cpp on dual RTX 3090 GPUs.
Issue Isolation
Previously, the author used --split-mode tensor, which yielded decent token generation (tg) speeds. However, he discovered that prompt processing (pp) was falling back to the CPU, bottlenecking the pp speed at around 400 t/s.
Performance Leap
Benchmarking revealed the hardware was capable of 1600+ t/s. By switching to --split-mode layer, the GPU took over sampling. While token generation slightly dropped from 60-70 t/s to 40-55 t/s, prompt processing skyrocketed by more than 4x, drastically improving overall long-context inference efficiency.
More from Infra
- How to Run FLUX Locally on RTX 5090? Reddit User Seeks Setup Advice — RutabagaSad8257 · 2026-08-07
- AWS Veteran's Deep Dive: Building Scalable Cloud Control Planes Over 14 Years — TheNickWalsh · 2026-08-07
- Musk: Terafab Texas Will Be the Largest and Most Valuable Building on Earth — beffjezos · 2026-08-07
- VIGIL: Real-Time Terminal Dashboard for Hardware Power Monitoring — tom_doerr · 2026-08-07
- ComfyUI Node Optimizes MiniMax H3 Generation Time by 30% — nnorbbi · 2026-08-07
- DARPA Launches ScAN Program to Develop Analog Neural Networks for AI Efficiency — inductionheads · 2026-08-07