Fixing llama.cpp Tensor Split Crashes on Multi-GPU Setups
_TheWolfOfWalmart_ · reddit · 2026-08-08
A developer shared a workaround for crashes when using tensor split in llama.cpp with multiple gfx1030 GPUs (like the V620).
- Root Cause: GPU memory corruption occurs when using the default microbatch size of 512 or higher.
- Solution: Set -ub 384 and set -b to a multiple of it, multiplied by the number of GPUs.
- Performance: Testing Qwen models on 2x V620 yielded 40-50+ t/s for the 27B Q80 model and 80-110+ t/s for the 35B-A3B Q80 model.
More from Infra
- Open-source ML Engineering Book massively updates GPU accelerator benchmarks — StasBekman · 2026-08-08
- Inference Performance Optimization: Visualizing P50 vs P90 Latency Drops — DanielLockyer · 2026-08-08
- Musk's Terafab: A $16.8B AI Chip Megafactory to Become the World's Largest Building — coinfanking · 2026-08-08
- llama.cpp RPC PR: Cuts 300GB Model Loading Time to 1.5 Minutes — Chuyito · 2026-08-08
- Counterintuitive Test: MiniMax H3 Full BF16 Model is Faster Than INT8 and Better at Physics — Wise_Revolution385 · 2026-08-08
- Nvidia to Invest Up to $3 Billion in Blackstone-Backed Power Firm — pstAsiatech · 2026-08-08