Running Qwen3.6 35B on a Single AMD R9700: vLLM Setup and INT4 Benchmarks
KriptacMessage · reddit · 2026-08-08
A developer shared optimization experiences and benchmark results for deploying Qwen3.6 27B and 35B models on a single AMD Radeon AI Pro R9700 GPU using vLLM.
Deployment Configuration
- Weights: Used INT4 quantized versions (W4A16), as the 35B FP8 model cannot fit into a single 32GB card.
- Parameters: Set tensor parallel to 1 and GPU memory utilization to 0.98. Enabled speculative decoding (numspeculativetokens=4) for the 27B model, which tested 17-48% faster than 8.
- Checkpoint Fix: Manually modify truncation.maxlength and padding in the INT4 repo's tokenizer.json to prevent vision features from breaking.
Performance
- 35B MoE: Achieved 7,800 tok/s prefill and 61 tok/s decode at 4K context; maintained 49.5 tok/s decode at 150K context.
- 27B Dense: Reached 59.6 tok/s decode at 4K context with speculative decoding, averaging an accepted length of 4.5.
More from Infra
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16
- DeepSeek V4.1 Flash on 4x DGX Spark: weights alone are 511GB, the 494 tok/s claim doesn't fit — Aroochacha · 2026-10-16
- Open-sourced Intel GPU Vulkan inference upgrades: 2x ingestion, 60%+ faster inference — Fun-Aardvark-1143 · 2026-10-16
- ELSFP pain points: >3dB insertion loss, costly PMF, harsh packaging requirements — jwt0625 · 2026-10-16