Homelab With 4x RTX 4090 Weighs vLLM+P2P Patch vs llama.cpp for Qwen Models
dowitex · reddit · 2026-09-07
A homelab user with four RTX 4090s (64GB RAM, Threadripper Pro, all PCIe x16) is choosing between two local inference setups for coding: (1) vLLM + Qwen 3.8 27B dense fp8 + 256K KV cache with a new open-source-driver P2P patch that only benefits vLLM, vs (2) llama.cpp + Qwen flash next MoE iq4xs + 8-bit 200K KV cache. vLLM lacks 4-bit quants and VRAM is too tight for fp8 flash next, so the question is whether flash next's extra intelligence outweighs vLLM's higher throughput.
More from Infra
- Perplexity goes local: private tasks hand off to on-device small models — HowDevelop · 2026-09-07
- DeepSeek V4 Flash at 75% off via Merge Gateway: $0.04/M input tokens through Sept 30 — shensi · 2026-09-07
- Qwen 3.8 Flash Next runs at 65 t/s on M3 Ultra, Q2 weights released on Hugging Face — ivanfioravanti · 2026-09-07
- The Myth of Self-Hosted AI: 'Local' Models Still Route Through the Cloud — nomad-nostalgia · 2026-09-07
- Could AI run entirely on your phone? It could upend OpenAI's pricing — kevinsurace · 2026-09-07
- Inference engineering is the underrated AI skill: KV cache, batching and p99 latency explained — techNmak · 2026-09-07