Jetson AGX 64GB at 50W Runs Qwen 27B at 10.5 token/s, Nearly Matching Framework Desktop
Boognevatz2 · reddit · 2026-10-17
A Redditor dug out two Nvidia AGX embedded boxes with 64GB RAM and recompiled llama.cpp for Gyro format support. At a 50W power envelope, the 108GiB TQ30 Qwen3.8-Flash-Next-Gyro hits 82 token/s prefill and 9.86 token/s generation; the 27B model runs at 8.3 token/s (Q8KXL) and 10.5 token/s (Q4KM) with MTP, while an MTP-preserved Qwen3.6-35B-A3B reaches 42 token/s. Comparing against Framework Desktop 128GB expectations (11-12 token/s), the author sees no order-of-magnitude upgrade within a $5,000 budget.
More from Infra
- Ben Horowitz says Amazon is sold out of compute through end of 2028 — RihardJarc · 2026-10-17
- Free 8-hour AI data center course launches on freeCodeCamp, from grid power to GPU economics — trawasthi_ai · 2026-10-17
- Broadcom Wi-Fi bug enables passwordless remote code execution; ~1.5M Belgian routers unpatched — jedisct1 · 2026-10-17
- Local LLM hosting isn't always cheaper: cached-token rate decides the break-even point — paq85 · 2026-10-17
- DeepSeek's new checkpoint cuts CoT token use by up to 10x, sparking Jevons paradox talk — teortaxesTex · 2026-10-17
- Latent Space: Meta's 816B Open Model Tops US Charts, TPUv7 Beats GB300 on $/perf — Latent Space · 2026-10-17