Running 35B Models on a Single RTX 5080: Migrating from llama.cpp to vLLM
McFlurriez · reddit · 2026-08-11
A developer is seeking advice on migrating their local LLM inference setup from llama.cpp to vLLM. Currently running a Qwen3.6-35B-A3B model on a single RTX 5080 using llama.cpp, they have to offload some layers to the CPU and are limited to a single parallel request (-np 1).
To support subagents in their coding harness, they want to switch to vLLM for better parallel request handling. However, they are finding vLLM's configuration complex and specifically ask how to replicate the -ncmoe 20 setting (for MoE expert offloading) from llama.cpp in vLLM.
More from Infra
- Jensen Huang: NVIDIA AI Factory Compute Is Becoming an Investable Asset Class — prasannaalahoti · 2026-08-11
- Training a 1B-Parameter LLM from Scratch for Just $200: Full Details Open-Sourced — SevereTilt · 2026-08-11
- Demand for AI Gateways and Model Routing Sees Explosive Growth — shensi · 2026-08-11
- OpenAI's Letter to Texas Governor on Responsible AI Infrastructure — borowcy · 2026-08-11
- Testing DGX Spark with H3: First Text-to-Video Run via ComfyUI — allaboutai-kris · 2026-08-11
- China's First Domestic Front-end Lithography Tool Enters Production, Competing with 90s ASML — pstAsiatech · 2026-08-11