GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU
TheZachMueller · x · 2026-10-28
Running GLM 5.3 Flash in NVFP4 quantization on four RTX 6000 Workstation cards power-limited to 450W, the author shares a V1-vs-V2 throughput comparison: per-agent output jumps from 140 to 240 tok/s at concurrency 1, and throughput per GPU from 115 to 210 tok/s. Even at concurrency 16, V2 sustains 165 tok/s per GPU versus 85 for V1 — the new serving stack roughly doubles efficiency on power-limited local hardware.
More from Infra
- Push-Based KV Transfer in vLLM Cuts Time-to-First-Token ~3x, Boosts TPOT Up to 30% — dl_weekly · 2026-10-28
- The data center boom has an upside: innovation for the US grid — TechCrunch AI · 2026-10-28
- DIY dual-RTX-5060-Ti rig benchmarks 12 local 4B-35B models with real wall-clock times — Careful-Report6526 · 2026-10-28
- Together GPU Clusters adds auto repair: from GPU fault to healthy node with no human in the loop — togethercompute · 2026-10-28
- Supabase launches Compute in private alpha: run AI agents and long-running jobs next to your Postgres data — dshukertjr · 2026-10-28
- Atreides' Gavin Baker: operating cash flow will fund more of the 2027 AI buildout than expected — BenBajarin · 2026-10-28