Running 27B Models on a Single RTX 5090: Multi-GPU vs. Edge Deployment Stats
Fz1zz · reddit · 2026-08-13
After acquiring an RTX 5090, a developer shared performance metrics and hardware advice for local LLM deployment.
- Main Rig: Running the Qwen3.6-27B model on a single 5090 with a 160k context, Flash Attention, and Multi-Token Prediction (MTP). The author argues that adding a 4070 Ti Super for a multi-GPU setup would likely slow things down rather than help, preferring to sell the old card.
- Edge Device: On a NucBox K8 Plus mini PC with an AMD Ryzen 7 8845HS, utilizing the iGPU and unified memory to run Gemma-4-26B, the setup achieved 343 t/s prefill and 31.4 t/s decode speeds.
More from Infra
- Tencent squeezes 770B-parameter Hunyuan Hy4 Preview into 214 GiB at ~2.38 bits per weight — coolmum18 · 2026-10-16
- TokenSpeed MLA 2 hits NVIDIA Vera Rubin: up to 3.44x faster decode vs B200 — zhyncs42 · 2026-10-16
- US vs Chinese hyperscalers: Americans fund and schmooze, Chinese demand your BoM and cap you at 7% margin — jwt0625 · 2026-10-16
- How LiteLLM routes one OpenAI-shaped API across every model provider — Mahmoud_Zalt · 2026-10-16
- OpenAI-Broadcom custom AI chip Jalapeño named a TIME Best Invention of 2026 — itsclivetime · 2026-10-16
- Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast — weskambale · 2026-10-16