Why Dual R9700s Lag Behind a Single RTX 5090: Debugging Local LLM Inference
TheyCallMeDozer · reddit · 2026-08-10
A developer shared their experience debugging a local LLM inference server built with dual AMD Radeon AI PRO R9700 GPUs (64GB total VRAM). When processing concurrent requests for the Qwen3.5 9B model, the server was significantly slower than a desktop with a single RTX 5090.
Troubleshooting & Observations:
- Concurrency Failure: Despite setting OLLAMANUMPARALLEL=4, logs showed Ollama was serializing requests, causing four concurrent calls to finish sequentially with increasing delays.
- Unbalanced GPU Utilization: rocm-smi revealed one GPU was fully saturated (100%) while the second sat completely idle (0%). The dual GPUs were only utilized together for larger models (e.g., 32B) requiring tensor splitting.
- High VRAM Footprint: Due to a massive configured context window of 262k, the KV cache consumed nearly 10GB, inflating the total memory footprint of the 9B model in Ollama to around 20GB.
- Underlying Errors: Logs showed a rocblaslt error indicating a failure to load a specific TensileLibrary file.
The author is seeking recommendations for alternative inference stacks (like vLLM or llama.cpp) to efficiently handle both high-throughput workloads (9B models) and large-context analysis (32B models).
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- Neocloud M&A decoded: CoreWeave, Nebius, Crusoe, Nscale all bet margins vanish beyond GPU scarcity — AccBalanced · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16