Fork brings near-Vulkan Metal LLM speeds to ancient AMD RX580 on Mac, 29 tok/s
opUserZero · reddit · 2026-11-03
The author released a fork of ToshLLM that brings Metal acceleration for llama.cpp to AMD GPUs (RX580/Polaris) on Intel Macs. Starting from 17 tok/s on Ornith 9b versus Linux Vulkan's 32 tok/s, tuning pushed it to 29 tok/s — usable on hardware this old (the CPU is 15 years old with no AVX).
Notably, GLM-5.3 Flash and Qwen3.8 Flash did most of the heavy tuning work while the author directed. They also mention a new Qwen 3.5 4b coder that can do 3D models, starting at 35 tok/s on their card, which they plan to try next.
More from Infra
- KVCache Meetup Korea dives into LLM serving: CXL pooling, roofline analysis, batch scheduling — PyTorch · 2026-11-03
- FuriosaAI's new round values it at 662x last year's sales, no chipmaker joined — YvesMulkers · 2026-11-03
- Just upgrading onnxruntime-gpu 1.23→1.31 with CUDA 13 made SAM 3 inference 3.8x faster — MaziyarPanahi · 2026-11-03
- Dev argues buying GPUs for local AI is just an expensive hobby — API and rented GPUs are always cheaper — Angaisb_ · 2026-11-03
- Local inference devs grapple with models breaking their own CoT on streamed tokens — SilentMobius · 2026-11-03
- FLAMIA: MXFP8 microscaling attention kernel for fast, causally-safe training on Blackwell — yresearch · 2026-11-03