Qwen dual-GPU inference optimization: 10x prefill speed boost achieved
Comrade_Mugabe · reddit · 2026-08-28
Benchmarking Qwen3.8-Flash-Next on dual RTX 3060s, the author found that llama.cpp's default -sm tensor mode causes MoE weights to fall back to CPU, capping prefill at 36 tps. Switching to -sm layer and tuning -ubatch 2048 increased prefill speed to 400 tps. The post compares ikllama.cpp, noting it lacks this trap and matches speed but consumes 75 GB more RAM. Detailed configs and metrics are provided for local deployment.
More from Infra
- Fluidstack hires Electrical Engineer for Modular R&D to build frontier AI infrastructure — MxMnr · 2026-10-16
- Jensen Huang joins BG2 podcast: does AI revenue justify capex, '27 bottlenecks — elonmusk · 2026-10-16
- Mistral's on-device infra rack revealed: DGX Spark, Mac Studio and more — gregmushen · 2026-10-16
- NVIDIA and Sweden's Wallenberg launch Sferical One, a shared AI supercomputer — nvidia · 2026-10-16
- Next infra frontier: self-evolving systems that adapt and self-verify — _shreya_s · 2026-10-16
- Xiaomi's MiMo RL dashboard reveals $3.47M spent on just 30 training steps — bycloud · 2026-10-16