Qwen 177B at 11-15 tok/s on a single RTX 5070 12GB via expert-streaming tweaks
ayobluestarr · reddit · 2026-10-03
A detailed writeup of running Qwen3.8-Flash-Next 177B (UD-IQ3XXS) on an RTX 5070 12GB with 32GB DDR4 and a Ryzen 5 5600GT using a llama.cpp expert-streaming setup: 11.5 tok/s in benchmarks (up from 7), 14-15 tok/s in normal chat, and a 4,892-token coding prompt at 10.15 tok/s producing a working single-file Snake game.
Key optimizations:
- Fixed Windows I/O queue-depth issues and used one file handle per worker
- Built a page-locked hot-expert tier so the GPU pulls hot expert weights efficiently
- Outputs quality-gated against a control model; benchmarks, methodology, and failed attempts are open-sourced on GitHub
The author is soliciting suggestions, especially around streaming.
Related event: RTX 5070 12GB Runs 177B Model Locally at 11-15 tok/s(2 posts)→
More from Infra
- Thailand data center association cites $18B in GPU/hardware cancellations after SE Asia lending halt — SumitGup · 2026-11-19
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19