GLM-5.3-Flash NVFP4 benchmarked: 2x H200 matches 4x RTX Pro 6000 on throughput
TheZachMueller · x · 2026-10-21
User @keennay ran llm-inference-bench on nvidia/GLM-5.3-Flash-NVFP4 (DFlash2 speculative decoding) across H200 and RTX Pro 6000 systems. Key numbers: 2x H200 hits 195 tok/s at 512k context (SGLang, c1) and 620 tok/s at 64k with 8 concurrent requests — nearly identical to 4x RTX Pro 6000 (167 / 620 tok/s), surprising given the lack of native FP4. RTX Pro 6000 peaks at 1,358 tok/s decode (8k ctx, 32 concurrency). Verdict: 4x H200 NVL is the sweet spot for NVFP4/MXFP4 LLMs at max context, especially when CPU memory offload matters (DeepSeek V4.1, GLM-5.3-Flash). 512k prefill still costs 46-55s TTFT.
More from Infra
- VectorDash Rewrites Windows Search in Rust, Adds Local Reverse Image Search and AI Chat — andy2na · 2026-10-21
- SpaceX earnings: compute buildout beats expectations, capex and debt remain the big risk — JOBhakdi · 2026-10-21
- Enterprise AI data pipelines should start with systems of record, argues Craig Mullins — craigmullins · 2026-10-21
- Musk claims SpaceX will add 200k GPUs monthly, targeting 13GW by 2027 — analysts skeptical — JOBhakdi · 2026-10-21
- SpaceX's AI business beats expectations as Q3 net income hits $4.4B — TechCrunch AI · 2026-10-21
- Together AI Berlin meetup: n8n and NVIDIA engineers on running open models in production — togethercompute · 2026-10-21