Optimizing GLM 5.3 Flash NVFP4 on a 450W RTX 6000 WS: 162 to 242 tok/s step by step
TheZachMueller · x · 2026-10-28
TheZach Mueller shares a stage-by-stage optimization log for serving GLM 5.3 Flash NVFP4 on power-limited (450W) RTX 6000 Workstations: baseline 162–168 tok/s/user, +P2P allreduce 201.5, +online NVFP4 attention 219.54, merged projection 220 median / 250 best, a KDA numwarps fix saving 0.12ms/round, and autotune + fused finalize reaching 242.70 @ 300s. A practical FP4 inference tuning ladder.
More from Infra
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- Will local-model progress outrun GPU depreciation? Reddit debates the buy-now window — Sensualities · 2026-10-28
- Dev cuts local AI inference power from 100W to 25W with only a 1.9s→2.8s slowdown — AryanEmbered · 2026-10-28
- SK Hynix posts $62.8B quarterly operating profit at 80% margin as AI memory demand booms — mhdfaran · 2026-10-28