Upgrading onnxruntime-gpu 1.23→1.31 made SAM 3 inference 3.8x faster on one L4
MaziyarPanahi · x · 2026-11-03
The author's malaria cell-counter got 3.8x faster without touching the model: same SAM 3, same L4 GPU, identical outputs (within 0.02px per box) — the only change was upgrading onnxruntime-gpu from 1.23 to 1.31 with its CUDA 13 libraries.
- Reproduced with 20 runs per slide in one container: 0.53s → 0.14s per word/inference
- Cause: the old build added 3,860 copy nodes to the graph; the new one adds just 8
- Takeaway: if your ONNX model feels slow on GPU, check the onnxruntime-gpu version first (fix suggested via @wkentaro's lockfile)
Related event: Upgrading onnxruntime-gpu Alone Speeds SAM 3 Inference 3.8x(2 posts)→
More from Infra
- Buyers underestimate on-prem AI hardware costs — expensive memory may be a cloud bull case — mobileraj · 2026-11-04
- World's biggest PCIe graphics card: prototype AMD board packs four RX 470 GPUs — chemist_slime · 2026-11-04
- Open-source Project NOMAD packs Wikipedia and a local AI server for offline use — drdanielbender · 2026-11-04
- 15-year-old's attention kernel survives 32k tokens where standard PyTorch OOMs, 11x faster — vankownlaw · 2026-11-04
- Five local models on DGX Spark: Qwen3.8-27B most reliable at 28/30, fastest models finish last — JeremyCMorgan · 2026-11-04
- Without Caching Your AI Agent Costs 10x More: Artificial Analysis Breaks It Down — AI Engineer · 2026-11-04