Running Qwen3.6 35B on a Single AMD R9700: vLLM Setup and INT4 Benchmarks

KriptacMessage · reddit · 2026-08-08

A developer shared optimization experiences and benchmark results for deploying Qwen3.6 27B and 35B models on a single AMD Radeon AI Pro R9700 GPU using vLLM.

Deployment Configuration

Weights: Used INT4 quantized versions (W4A16), as the 35B FP8 model cannot fit into a single 32GB card.
Parameters: Set tensor parallel to 1 and GPU memory utilization to 0.98. Enabled speculative decoding (numspeculativetokens=4) for the 27B model, which tested 17-48% faster than 8.
Checkpoint Fix: Manually modify truncation.maxlength and padding in the INT4 repo's tokenizer.json to prevent vision features from breaking.

Performance

35B MoE: Achieved 7,800 tok/s prefill and 61 tok/s decode at 4K context; maintained 49.5 tok/s decode at 150K context.
27B Dense: Reached 59.6 tok/s decode at 4K context with speculative decoding, averaging an accepted length of 4.5.

Original post →

Running Qwen3.6 35B on a Single AMD R9700: vLLM Setup and INT4 Benchmarks

More from Infra