Running Qwen3 27B agent on 16GB VRAM: MTP draft trades context and quality for speed
Remarkable_Air_8383 · reddit · 2026-10-03
The author serves a local Hermes agent with llama.cpp, running a Qwen3 27B model (iq3s quant) on 16GB VRAM. With MTP drafting enabled, prefill slows down and the model seems less capable, and tight VRAM forces context down to 96k while decode hits 40-60 tps. Disabling MTP restores 128k context but decode drops to 35 tps; the post asks which trade-off to pick.
More from Infra
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Leak Hints at Classified 1M-GPU Training Cluster in China That Skips HBM — teortaxesTex · 2026-11-19