Running Qwen3.8-27B EXL3 on RTX 3060 + 5060 Ti: 50 tok/s with tensor parallelism and MTP
bring_back_the_v10s · reddit · 2026-09-21
A user details running Qwen3.8-27B EXL3 (5.0bpw) on a mixed-GPU rig — RTX 5060 Ti 16GB + RTX 3060 12GB (28GB total), 32GB RAM, Ryzen 5 5600, Linux — via exllamav3/tabbyAPI.
Key setup and results:
- Native tensor parallelism with auto GPU split, 102K context, Q8 cache;
- MTP draft mode lifts throughput to 50 tok/s average (40-60 swing) vs 22 tok/s without;
- Pitfalls: manual GPU-split tweaks hang the engine; 1GB VRAM used by the desktop environment contributes to imbalance (leftover: 1.3GB on the 3060, 0.6GB on the 5060);
- He already used it to write a small Rust TUI app with ratatui.
Full config.yml included — directly reusable for anyone running 27B models on older cards.
More from Infra
- Activist evicted in 2024 puts $5B Maryland AI data center on hold — ATTlKA · 2026-11-12
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12