Strangely, GPU matmuls run faster on 'predictable' data: Horace He explains
goyal__pramod · x · 2026-09-25
Horace He recounts a counterintuitive finding: CUTLASS's profiler showed 288 TFLOPS vs CuBLAS's 258 on an 8192³ matmul, but the gain vanished in Python. Ablations revealed the profiler initializes inputs with integers only — zeros hit 295 TFLOPS while randn inputs drop to 257. The values' distribution directly affects matmul runtime, and the post explores why from first principles.
More from Infra
- GPU returns hit +76% a year as H100 rents jump 49% and B300 rates climb 66% — KyeGomezB · 2026-09-25
- openjev-sglang: SGLang Radix Cache lets Qwen3.6-35B-A3B run 64 Jev decisions in under 1s — multiply_matrix · 2026-09-25
- Ramp benchmarks Jev to replace LLM reranking: 10x lower tail latency at 300ms, 3x cheaper — multiply_matrix · 2026-09-25
- Qualcomm pitches the phone as the AI hub at Snapdragon Summit, aiming for Apple-like cross-device experience — BenBajarin · 2026-09-25
- Running Qwen3.8-Flash-Next on a 5090 with llama.cpp: 40 tok/s and barely any RAM used — nirurin · 2026-09-25
- Exelon refuses to power $20B hyperscaler data center after developer pays $1 deposit — SumitGup · 2026-09-25