Why Dual R9700s Lag Behind a Single RTX 5090: Debugging Local LLM Inference

TheyCallMeDozer · reddit · 2026-08-10

A developer shared their experience debugging a local LLM inference server built with dual AMD Radeon AI PRO R9700 GPUs (64GB total VRAM). When processing concurrent requests for the Qwen3.5 9B model, the server was significantly slower than a desktop with a single RTX 5090.

Troubleshooting & Observations:

Concurrency Failure: Despite setting OLLAMANUMPARALLEL=4, logs showed Ollama was serializing requests, causing four concurrent calls to finish sequentially with increasing delays.
Unbalanced GPU Utilization: rocm-smi revealed one GPU was fully saturated (100%) while the second sat completely idle (0%). The dual GPUs were only utilized together for larger models (e.g., 32B) requiring tensor splitting.
High VRAM Footprint: Due to a massive configured context window of 262k, the KV cache consumed nearly 10GB, inflating the total memory footprint of the 9B model in Ollama to around 20GB.
Underlying Errors: Logs showed a rocblaslt error indicating a failure to load a specific TensileLibrary file.

The author is seeking recommendations for alternative inference stacks (like vLLM or llama.cpp) to efficiently handle both high-throughput workloads (9B models) and large-context analysis (32B models).

Original post →

Why Dual R9700s Lag Behind a Single RTX 5090: Debugging Local LLM Inference

More from Infra