GLM-5.2 local inference: ubatch size significantly boosts MoE performance
fuzhongkai · reddit · 2026-08-22
Tests on GLM-5.2-UD-IQ2XXS GGUF across 3x RTX PRO 6000 Blackwell GPUs reveal that larger ubatch sizes significantly improve throughput for long prompts (763 to 1145 t/s) but not short ones. The author attributes this to the 256-expert / top-8 MoE architecture, where larger batches better utilize expert GEMMs. Additionally, layer splitting outperforms Tensor Parallelism (TP=3) on PCIe without NVLink.
More from Infra
- Fluidstack hires Electrical Engineer for Modular R&D to build frontier AI infrastructure — MxMnr · 2026-10-16
- Jensen Huang joins BG2 podcast: does AI revenue justify capex, '27 bottlenecks — elonmusk · 2026-10-16
- Mistral's on-device infra rack revealed: DGX Spark, Mac Studio and more — gregmushen · 2026-10-16
- NVIDIA and Sweden's Wallenberg launch Sferical One, a shared AI supercomputer — nvidia · 2026-10-16
- Next infra frontier: self-evolving systems that adapt and self-verify — _shreya_s · 2026-10-16
- Xiaomi's MiMo RL dashboard reveals $3.47M spent on just 30 training steps — bycloud · 2026-10-16