Cosmos3-Edge fits in 8GB: INT4 quantization triples decode speed on Jetson
No-Dragonfly6246 · reddit · 2026-10-21
A developer got NVIDIA's Cosmos3-Edge running on the 8GB Jetson Orin Nano Super—previously impossible with the BF16 stack.
Key details:
- With INT4 quantization plus speed optimizations, the Orin Nano Super hits 152.5 decode tok/s, vs 44.1 tok/s for BF16 vLLM on the larger AGX Orin 64GB; AGX Thor 128GB reaches 207.8 optimized (68.2 BF16).
- Accuracy across five VLM benchmarks barely drops: 65.56 → 65.20; optimized VLM peak memory goes as low as 3.3GB.
The VLM portion is published now; diffusion (video gen) and robot policy parts are coming soon. The author is also testing a page for pre-compiled models since Hugging Face doesn't support them.
More from Infra
- One researcher pretrains a frontier-class model in 3.5 months for under $10M at Mirendil — vjkaruna · 2026-10-22
- Crusoe unveils Tinker API-compatible LLM post-training platform at PyTorch Conference — PyTorch · 2026-10-22
- SGLang-TorchTPU launches with Google and Meta, hits 770 tok/s decode in production — ying11231 · 2026-10-22
- Open PR optimizes DeepSeek-V4.1 inference on RTX PRO 6000 Blackwell — AlpinDale · 2026-10-22
- vLLM adds day-0 support for Flint: 500 TPS on a single GB200 — _rockt · 2026-10-22
- Grass founder on the Delphi Podcast: training data is now a profitable business serving top labs — grass · 2026-10-22