GTX 1080 Ti + MI50 Vulkan llama.cpp benchmarks: small MoE hits 640 t/s prefill
tabletuser_blogspot · reddit · 2026-09-15
A Reddit user benchmarked 13 models on llama.cpp's Vulkan backend using a mixed Nvidia GTX 1080 Ti (11GB) + AMD Instinct MI50 (16GB) setup on an i9 36-core system, with Flash Attention enabled (negligible impact).
Key results:
- Small MoE models shine: bailingmoe2 16B.A1B hit 643.73 t/s prefill and 123.29 t/s generation, the fastest overall
- qwen3moe 30B.A3B reached 295/62 t/s; qwen35moe 35B.A3B MXFP4 hit 332/20 t/s
- Sparse MoE generation far outpaces dense models of similar size (27B dense: only 15-17 t/s)
- A 4.5 bpw-quantized Phi-3.5-MoE (42B) still generates at 50 t/s
Takeaway: mixed-vendor old GPUs + Vulkan + small MoE models make for a surprisingly usable local inference rig.
More from Infra
- Prefill and Decode: why asking an LLM for three takeaways from a long document still takes minutes — dotey · 2026-09-15
- Training from scratch on a single H100 hits 76% on ARC-AGI-1 in ~4 hours — GregKamradt · 2026-09-15
- From Ollama to vLLM: a roadmap for scaling LLM deployment — kalyan_kpl · 2026-09-15
- Kimi K3 is live and free on NVIDIA NIM with OpenAI-compatible API — airesearch12 · 2026-09-15
- Dual Radeon AI Pro R9700 vs. Two Used RTX 3090s at $1600 Each for Local LLM Inference — Current-Ticket4214 · 2026-09-15
- RTX 5090 doubles to $6,899 as RAM prices eclipse GPUs in worst-ever PC build market — Yuchenj_UW · 2026-09-15