Jetson Thor LLM inference hits only ~60% of memory-bandwidth estimate
resrev3R · reddit · 2026-11-19
A Reddit user compared LLM inference on Jetson Thor vs RTX 5070 Ti and found an unexpected gap:
- 5070 Ti (896 GB/s) running Qwen 27B Q4: bandwidth-based estimate 66 tok/s matches reality
- Jetson Thor (273 GB/s) should theoretically hit 20 tok/s but only manages 12 tok/s — about 60% of the estimate
- Surprisingly, DFlash, which gives big gains on discrete GPUs, delivers less than 2x speedup on Thor, barely better than MTP
The author assumed memory bandwidth was the main inference bottleneck, but Thor's results suggest another limiting factor, and is asking for explanations.
More from Infra
- Micron's Scott DeBoer says 3D DRAM is coming in 2028/2029 — firstadopter · 2026-11-19
- Two Energy Clocks for AI Infrastructure: Flexibility Buys Time, Capacity Sets Scale — ingliguori · 2026-11-19
- NVIDIA to demo Cosmos Data Factory pipeline for training-ready synthetic datasets with Voxel51 and Encord — NVIDIAAI · 2026-11-19
- Analyst: memory has shifted from commodity to value, explaining sustainably higher margins — BenBajarin · 2026-11-19
- Microsoft open-sources Aion 2.3B on-device SLM, shipping as the Windows built-in model — jacek2023 · 2026-11-19
- Local model reads a 131K-token patient history in 61s, then reasons for 11 minutes — maniac518 · 2026-11-19