Kipply breaks down transformer inference arithmetic for H200/B200 in new perf engineering repo
ycombinator · x · 2026-09-29
waferai launched what it calls the most comprehensive AI performance engineering repo, with part 6 by Carol Chen (kipply) covering Transformer Inference Arithmetic.
- Builds an approximate inference cost model around per-token compute, bytes moved by the GPU, and inter-GPU communication, as a starting point for latency/throughput analysis on H200, B200, and B300.
- Examples use A100 GPUs, combined with NVIDIA Hopper and Blackwell documentation.
- Key takeaway: prefill and decode expose different token parallelism — prefill processes prompt tokens together for more weight reuse in projection/FFN matmuls, while small-batch decode can spend more time loading weights than computing. Profile each phase separately.
- Batching raises arithmetic intensity (thread truncated in source).
More from Infra
- Sarvam AI Launches Token Factory: India-Hosted Inference for Open Models — itsOmSarraf_ · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16