$16K CAD 4x R9700 local inference server: 190 tok/s per user, 8 users at 128K context benchmarked
Anarchaotic · reddit · 2026-11-09
A Reddit user built a local multi-user inference server with 4x AMD Radeon AI PRO R9700 (CAD $16K total) to run Qwen3.8-Flash-Next on sensitive business data.
Setup: EPYC 7402P, 128GB DDR4, Radiance v1.17 inference stack, tp4, FP8 KV cache, MTP speculative decoding, cards power-limited to 210W.
Early benchmarks:
- 190 tok/s write speed for a single user; total prefill 6,800 tok/s regardless of user count.
- With 8 users at 128K context each, agent follow-ups start in 0.2–2.5s and write at 80–225 tok/s per user.
- Worst case (8 simultaneous fresh 128K prompts) waits 2.5 min for first token.
- 17.4 GiB weights per card; 47.7 GiB n-gram/PLE table pinned in host RAM; 11.6 GiB KV cache per card = 1.6M tokens.
Much quieter than expected and a big step up from his previous Strix Halo 395+128GB.
More from Infra
- Vork: an agent-native 3D engine that renders 24x faster than Blender with a 2,500-token API — ravisparikh · 2026-11-09
- Akamai Edges Cloudflare 78-76: A 13x Sales-Multiple Gap Between Two Edge Giants — pdamodaran · 2026-11-09
- Out of Claude and Codex tokens, dev spins up local vLLM serving Qwen FP4 model for agents — alfcnz · 2026-11-09
- AGI House's personal agents build day pushes local inference and intelligence-per-watt — agihouse_org · 2026-11-09
- Agentic AI is orchestration of model calls; datacenter limits may push edge models — DavidLinthicum · 2026-11-09
- "Bro got distilled": meme pairs Jensen Huang photo with Huawei's Nvidia chip challenge — StephanSturges · 2026-11-09