NVIDIA's open-source AIPerf benchmarking tool and Triton GPU tutorials for inference measurement
techNmak · x · 2026-10-17
techNmak continued his free inference-engineering resource list with two picks:
- NVIDIA AIPerf (open source, Apache-2.0, 791 GitHub stars): a comprehensive benchmarking tool measuring first-token latency, token latency, throughput, and goodput for generative AI models, with guides for controlled loads and realistic workload traces. Basic Python, PyTorch, and Transformer knowledge recommended
- Triton official GPU programming tutorials (posted as a reply): runnable examples going from vector addition to softmax, matrix multiplication, and fused attention for learning how GPU kernels work
Learning materials are free; running examples may require paid compute.
More from Infra
- MiniCPM-V 4.7 runs locally at 100 tok/s using only ~3 GB of memory — NielsRogge · 2026-10-17
- Power-management chips: the invisible layer deciding AI data center energy efficiency — Olivier__OG · 2026-10-17
- MiniCPM-V 4.7 runs a 35B multimodal model in 3GB of memory, fully on-device — alifcoder · 2026-10-17
- Full FP8 n-gram table visibly improves local Qwen3.8-Flash-Next at zero speed cost — sillynoobhorse · 2026-10-17
- Local Nemotron 3.5 on one A100 hits 169 tok/s, processes 100 clinical notes for 2.4 cents — MaziyarPanahi · 2026-10-17
- Official Triton GPU Programming Tutorials: Vector Add to Fused Attention — techNmak · 2026-10-17