Running Qwen3.8 27B with 262k context, q4_0 KV and vision on 16GB VRAM at 80t/s
Fancy-Snow7 · reddit · 2026-10-13
Reddit user Fancy-Snow7 details running unsloth/Qwen3.8-27B-UD-IQ4XS on a single 16GB card (RTX 5070 Ti) with 262k context, q40 KV cache, vision enabled, 80t/s MTP and 1500pp — while also driving dual monitors. The trick: the llama.cpp-adaptive-kv-streaming fork, modified by the author to support q40 KV (a few lines of code), with KV overflow streamed through a 1.5GB kv-stream-arena from system RAM. Full-context speed drops to 20t/s (pp 400t/s); the author compacts around 180-200k. A simpler alternative is q80/q40 KV with batch 128 at 100t/s pp cost. Caveat: watch other VRAM-hungry apps, as Windows spillover tanks performance.
More from Infra
- HF's Zach Mueller: don't do this — most users can just rent or run sandboxes locally on CPUs — TheZachMueller · 2026-10-13
- Celesto CI undercuts GitHub Actions 9x: 1000 jobs for $6.67 vs $60 — aniketmaurya · 2026-10-13
- Keurig coffee maker flooded Wi-Fi with 1TB in 10 days — owner says unit bug — gnukeith · 2026-10-13
- oQe Quantization Update Cuts Code KL Divergence by Up to 18% Across 69 Models — huggingface · 2026-10-13
- Rodolfo Rosini on 2040: cheap compute everywhere, devices jumping orders of magnitude — MikePFrank · 2026-10-13
- AI inference chips are a $225bn market: Vaire on why startups avoid silicon and its Nvidia-adjacent plan — MikePFrank · 2026-10-13