Open-source fork runs 47GiB MoE on 2019 Radeon VII with 49% faster decode
milpster · reddit · 2026-11-07
Developer milpster released a strata fork bringing gfx906 (Radeon VII/MI50) support and mixed AMD+NVIDIA inference, aimed at ROCm-primary systems with an idle NVIDIA card sitting around.
- gfx906 actually works: full HIP backend plus a custom single-target rocBLAS replacing the stock library that segfaults on first inference.
- Cross-vendor serving: the NVIDIA card serves expert/prefill rows to the AMD engine over a socket — no stock cross-vendor path exists.
- Speculative decoding on HIP: DFlash2 drafter runs on AMD cards, with exact-match verification for sampled requests.
- Self-healing pair: SIGKILL the helper and requests still finish; the supervisor rebuilds automatically.
- Numbers: pooled median decode 34.7 vs 23.3 tok/s (+49%), 1.4–2.4× faster 200–600 token reads, −6.2 GiB host RAM; honest cost of −12% on fresh 16k prefill reads.
- Quality proven: greedy 80k-token infill byte-identical to stock, teacher-forced gate top-1 100% / KL 0.00000, 291 tests green; every speed win gated by both speed and quality lanes.
Repo includes a pinned ROCm 6.1 build guide, A/B/C-vs-stock bench harness, and full session journal — every claim re-runnable.
More from Infra
- Underdog Woof: a 31.7 MB TTS model that runs on CPU 13.8x faster than real time — victormustar · 2026-11-07
- LLM inference flips between compute-bound and memory-bound within a single request — techNmak · 2026-11-07
- Four TTS models small enough to run on CPU, smallest just 1.4M params — vanstriendaniel · 2026-11-07
- MiniMax H3 Runs Locally on a RTX 5060 Ti 16GB — Full ComfyUI Setup Shared — Description-Serious · 2026-11-07
- AI chip stocks rebound after steep fall, but AMD and Nvidia recovery may be temporary — emmanuelvivier · 2026-11-07
- Counterpoint to Altman: a fully loaded H100 is a space heater that does math — thomasahle · 2026-11-07