Just upgrading onnxruntime-gpu 1.23→1.31 with CUDA 13 made SAM 3 inference 3.8x faster
MaziyarPanahi · x · 2026-11-03
Maziyar Panahi shares a free win: his malaria detection pipeline (SAM 3 ONNX model on an L4 GPU) got 3.8x faster over the weekend with zero model changes — just upgrading onnxruntime-gpu from 1.23 to 1.31 with its CUDA 13 libraries.
A/B benchmark (20 runs per word): "cell" outlines median 0.529s → 0.139s; total 1.06s → 0.28s for both words, with every bounding box identical to 0.0 px. Takeaway: if your ONNX model feels slow on GPU, check the runtime version first.
More from Infra
- Report: DeepSeek V5 to be pretrained almost entirely on Huawei Ascend chips, shunning Nvidia — kimmonismus · 2026-11-03
- Linus runs Qwen3 27B in LM Studio, gets roasted for skipping vLLM's 5x speed — victormustar · 2026-11-03
- SK Hynix subsidiary Solidigm plans $100B pair of NAND fabs in New York — zephyr_z9 · 2026-11-03
- Defer: a fully deterministic LLM inference engine with bit-for-bit reproducible output — iyioioio · 2026-11-03
- KVCache Meetup Korea dives into LLM serving: CXL pooling, roofline analysis, batch scheduling — PyTorch · 2026-11-03
- FuriosaAI's new round values it at 662x last year's sales, no chipmaker joined — YvesMulkers · 2026-11-03