Nemotron 3.5 Super VL: NVFP4 on one Blackwell GPU beats BF16 on two, ~2.2x per-GPU throughput

TheZachMueller · x · 2026-10-22

Lambda benchmarked NVIDIA Nemotron 3.5 Super VL (120B-A12B) on SGLang:
NVFP4 · 1× Blackwell: 1,860 tok/s peak system throughput (C16), 587 tok/s/user single-stream;
BF16 · 2× Blackwell: 1,669 tok/s peak, 530 tok/s/user single-stream.
On the same 8,192-in/1,024-out workload, NVFP4 on one GPU delivers 2.2x the per-GPU throughput of BF16 on two B200s, since Blackwell runs FP4 natively. Both paths use multi-token-prediction EAGLE speculative decoding (5 steps, 6 draft tokens).
Caveat: decode CUDA graphs are captured only up to batch 16, so throughput regresses past C16; the useful operating range is C1–C16.

More from Infra