NVIDIA shows how to run Gemma and Qwen locally on Jetson with Ollama and vLLM
NVIDIA Developer · youtube · 2026-07-29
NVIDIA’s Jetson AI Lab session shows how to run open-source GenAI locally on Jetson devices without cloud access.
- It walks through getting models like Gemma and Qwen running end to end on-device.
- The stream compares Ollama for rapid prototyping, vLLM for higher-throughput serving, and llama.cpp for lightweight edge deployment.
- It also covers Jetson-specific serving choices for Orin and Thor, plus practical tuning topics like quantization and speculative decoding.
- A live demo of Reachy Mini shows the stack running in real time on edge hardware.
More from Infra
- Meta's commitments jumped to $349B in a quarter as server memory prices tripled ahead of Oct 21 earnings — tengyanAI · 2026-10-16
- Meta releases Muse Spark Open, trained with vLLM-powered RL inference from day one — colinmcnamara · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16