Routing NVIDIA PAIR to llama.cpp on an AMD ROCm node (2×R9700): full notes
Don_Reuter · reddit · 2026-09-11
A detailed writeup of making NVIDIA's Personal AI Router (PAIR) route to a tuned llama.cpp setup on an AMD ROCm node. PAIR ships GPU telemetry only for NVIDIA nodes, so the author added AMD telemetry via amd-smi (ROCm 7.2.0) and fronted an existing llama-swap server with a custom process-mode engine manifest + adoption. Verified on 2×R9700 (gfx1201) under the Go race detector. Code was vibe-coded by a local Qwen model with local-model review. Gotchas: manifests are embedded at build time (no rebuild = no effect), proxies are per-protocol not per-node, and the Intel xpu-smi changes are untested. A practical guide for heterogeneous local-inference clusters.
More from Infra
- Stanford and Together AI paper: hybrid local-cloud routing cuts AI cost and energy by 60-80% — rohanpaul_ai · 2026-09-11
- Rented GPU Bills: Host CPU and Script Defaults Made Costs 31x Higher — Worldly_North_7213 · 2026-09-11
- Running Qwen3.8 locally on a 128GB laptop for agentic coding: thinking tokens, not tok/s, set the wall clock — deepu105 · 2026-09-11
- SpaceX plans to make scarce turbine parts as AI data centers outpace gas turbine supply — rohanpaul_ai · 2026-09-11
- AGI as task time horizon vs meetings — and why fabs should train their own models — jwt0625 · 2026-09-11
- Free client-side calculator compares LLM token economics across DeepSeek, Claude, o3-mini — nikola_mr64990 · 2026-09-11