$1400 decade-old V100 32GB runs Qwen 27B: Q4_K_M +48% gen speed, Q8_0 +16% prefill
PuzzleheadedArm3732 · reddit · 2026-11-22
A Redditor tested Qwen3.8-27B quantization levels on a $1400 decade-old setup (Dell Precision 5820, 64GB DDR4, V100 PCIe 32GB capped at 225W), fully in VRAM with stock llama.cpp: Q4KM generates 48% faster than Q80, while Q80 prefills 16% faster (10% at 32k context); deep context costs 15% generation speed overall (Q4KM slides from 35.5 to 29.5 tok/s at 32k). For reference, Qwen3.5-35B-A3B (MoE) hits 104 tok/s on the same box but is less capable than the 27B dense. Verdict: the V100 remains very usable.
More from Infra
- Intruder swapped Prime Intellect's hosted DeepSeek for an abliterated copy for 53 hours — mawcopolow · 2026-11-22
- Token economics model projects $1.14T global API spend by 2030, but only 1% of users pay beyond coding — AccBalanced · 2026-11-22
- Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses — PathfinderTactician · 2026-11-22
- llama.cpp Vulkan SCALE fix PR #32735 delivers 2.3-3.2x speedup on Radeon R9700 — pmttyji · 2026-11-22
- Inference Stack Launches One-Command Self-Hosted LLM Serving with vLLM, LiteLLM and llm-d — AccBalanced · 2026-11-22
- Krea-2-Turbo runs on a GTX 1650 Super with 4GB VRAM at ~30s per image — A_NASHEX · 2026-11-22