Radeon AI PRO R9700 user gets 20 tok/s on Gemma 4-26B and suspects missing MoE tuning
veryhasselglad · reddit · 2026-07-23
vLLM on AMD R9700 tops out at 20 tok/s in a Gemma 4-26B test
A Reddit user reports only 19–20 tokens/s on a single Radeon AI PRO R9700 (gfx1201) running cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit with vLLM on ROCm, well below the 50 tok/s they expected from a published result.
The post includes the main runtime details:
- vLLM 0.22.1rc1.dev499+g470229c37.d20260613
- TP=1
- max-num-seqs=1
- max-model-len=8192
- gpu-memory-utilization=0.90
- kv-cache-dtype=fp8
- attention backend=TRITONATTN
- --enforce-eager
The user also sees a startup warning about a missing tuned MoE config for E=128,N=704,devicename=AMD-gfx1201,dtype=int4w4a16.json, and asks whether that missing config or --enforce-eager is the main reason for the slowdown. They also note Lemonade’s portable gfx120X runtime fails on this card because it ships gfx1200 kernel packs.
Overall, it is a practical troubleshooting thread about AMD inference performance, MoE tuning, and ROCm/vLLM benchmarking.
More from Infra
- AMD releases Lemonade 11.5 with a completed Lemonade Router — Fcking_Chuck · 2026-07-23
- Image generation slows from 20 seconds to over a minute after 20–30 batches — Greyblades2 · 2026-07-23
- Ben Bajarin says AWS and Azure may look much closer on next week’s cloud chart — BenBajarin · 2026-07-23
- Compute may matter more than we think at every layer of the AI stack — rakyll · 2026-07-23
- Inference at scale is the real inflection point, Ben Bajarin says — BenBajarin · 2026-07-23
- Google’s margin hit could favor infra vendors that let it bring its own stack — BenBajarin · 2026-07-23