Two 4060s + AMD Strix via RPC offload: llama.cpp context jumps 150k→195k at no cost
Chuyito · reddit · 2026-10-16
A hardcore llama.cpp speculative-decoding setup: main weights run on 2x RTX 4060 in tensor parallel, while a patched llama-rpc offloads the drafter to an AMD Strix iGPU — freeing 2-3GB VRAM at zero performance cost and extending context from 150k to 195k.
The key breakthrough: mixed Nvidia+AMD via RPC previously forced layer-only split mode. By merging PR #26636, PR #27858, plus a local patch (forcing the DFlash2 CPU selector to bypass the vocab-split lmhead limitation under tensor parallelism), the author kept split-mode tensor working.
Reproducible details:
- Known-good llama.cpp base commit and full cherry-pick/patch commands provided
- Missing 10Gbps USB NIC drivers solved by having GPT modify two source files for the right OS build
- The third fix isn't worth a PR (upstream PRs go dormant), so it's shared as a patch agents can rebase onto current main
- In theory anything with 2-3GB spare can serve as the drafter target
More from coding & agent
- Garry Tan endorses Aside Browser as the fix for agents getting blocked on the web — garrytan · 2026-10-23
- 1Password launches tool letting AI agents log in for you without seeing your password — AndrewSchmidtFC · 2026-10-23
- MCP server turned rogue 30 days after approval, prompting open-source mcp-guard lockfile tool — ErgoniaWorks · 2026-10-23
- Test: Coding Agents Won't Use an MCP Tool Unless AGENTS.md Mentions It — colinmcnamara · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- awesome-ai-apps: 132 Free Open-Source AI Agent and RAG Projects on GitHub — Arindam_1729 · 2026-10-23