llama.cpp Adds Cost-Based Tensor Split for 3-4% Speedup on Hybrid Multi-GPUs

milpster · reddit · 2026-08-11

A developer introduced a new cost-based tensor split mode (-sm cost) to llama.cpp, delivering significant inference speedups on specific multi-GPU configurations.

Core Improvement: The default layer split treats all layers equally, but in hybrid models (e.g., Qwen3.6 with Mamba + Attention layers), attention layers are computationally much more expensive. The new mode weights attention layers at 4.0 and Mamba layers at 1.0 for smarter load balancing.
Performance Gain: In mixed-speed multi-GPU setups (e.g., AMD + NVIDIA), this reduces the workload on the slowest GPU by 1-2 layers. Combined with pipeline parallelism, it yields a consistent 3-4% Token Generation (TG) speedup.
Topology Advice: The author highlights the bottleneck of cross-vendor GPU transfers bouncing through system RAM. A "fast-slow-fast" device ordering is recommended, placing the slow GPU in the middle to overlap cross-vendor transfer latency with computation.
Use Case: This optimization specifically benefits hybrid models (Mamba/SSM + Attention) running on mixed-speed GPUs with pipeline parallelism enabled.

Original post →

llama.cpp Adds Cost-Based Tensor Split for 3-4% Speedup on Hybrid Multi-GPUs

More from Infra