NVIDIA open-sources TensorRT Model Connect, two commands to TensorRT inference
JFPuget · x · 2026-08-19
NVIDIA open-sourced TensorRT Model Connect (Public Preview): supported HuggingFace models can reach end-to-end TensorRT inference in two commands, no intermediate ONNX export, with the bundle runnable via native C++ APIs. The entire project—implementations, perf tuning, tests, integrations, docs—was built with OpenAI Codex agents, with humans directing and reviewing.
TVM creator Tianqi Chen highlighted first-class TVM-FFI support, letting custom DSL and agent-generated kernels run inside TensorRT. The official tutorial shows replacing nodes in a Qwen3-8B graph with a TVM-FFI kernel and swapping a DistilBERT region with a CuTe DSL kernel—no TensorRT C++ required. Two paths exist: a recommended path using family-published versioned recipes, and an advanced path with manual graph inspection and node-ID typing. Both share the same backend; DSOs are bound at pipeline construction and can be swapped for ABI-compatible ones without rebuilding the bundle.
Related event: NVIDIA Open-Sources TensorRT Model Connect for Two-Command HF Deployment(2 posts)→
More from Infra
- First AA-AgentPerf results: NVIDIA Vera Rubin NVL72 delivers up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- Vera Rubin NVL72 first benchmarks: up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16
- 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16
- Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
- Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16