humans& W4A16 kernel hits 98% Tensor Core utilization, 10% faster than FlashInfer NVFP4
niloofar_mire · x · 2026-10-24
The humans& team showed that keeping weights at 4-bit while raising activation precision from 4 to 16 bits makes their NVFP4 decode kernel up to 25% faster than FlashInfer's optimized NVFP4 MegaMoE, yielding up to 10% more generation throughput and 7% lower overall latency with better accuracy — the opposite of the usual speed-vs-accuracy trade-off.
A shared chart shows 98.32% Tensor-pipe activity vs 57% for trtllm-gen BF16 W16A16. In SGLang serving benchmarks at concurrency 8, the work delivers 10% higher throughput and 7% lower TPOT. It's already used in Ampersand's RL training and Boba serving, with innovations flowing back to the open-source community.
More from Infra
- Grok does the math: running Kimi K3.1 Pro "locally" takes 18-32 H100s and $0.5-2M in hardware — giffmana · 2026-10-24
- A ~€1600 local LLM rig: i7-14700KF + RX 9070XT hits 70-90 tokens/sec — Snoo-2768 · 2026-10-24
- Jensen Huang: Nvidia software licensing could add ~$20B/year at $2,000 per enterprise GPU — sudoraohacker · 2026-10-24
- The 'compute spread': H100 rental vs open-source LLM token prices show -0.46 correlation — AccBalanced · 2026-10-24
- Neocloud premium is a speed premium: why Microsoft pays CoreWeave despite 5% borrowing costs — AccBalanced · 2026-10-24
- On-demand GPU rental indices hit all-time highs as even A100 prices rebound — AccBalanced · 2026-10-24