NVIDIA Tutorial: Fine-tune Models on Jetson with Unsloth
danielhanchen · x · 2026-09-01
NVIDIA Jetson AI Lab released a tutorial demonstrating how to fine-tune and quantize AI models directly on edge devices like the Jetson series. The workflow integrates Unsloth and memory-efficient QLoRA to export GGUF files for local inference via llama.cpp.
Key Highlights:
- Requirements: JetPack 7.2, Python 3.12, and significant NVMe storage (40GB for Qwen, 270GB for Nemotron).
- Workflow:
- Fine-tune: Adapt models using QLoRA.
- Convert: Export adapters to GGUF format.
- Run: Execute local GPU-accelerated inference with existing GGUFs.
- Examples: Includes hands-on steps for Qwen3.5-4B on Jetson Orin Nano and NVIDIA Nemotron 3.5 Lightning on Jetson AGX Thor.
More from Infra
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16