Optimizing Qwen 27B on Single RTX 3090: Quantization and Context Settings
randomjapaneselearn · reddit · 2026-08-26
A user shared their experience optimizing Qwen 3.8 27B on a single RTX 3090 using llama.cpp.
Background:
- Migrated from Qwen 3.6 35B-A3B due to high context consumption on simple tasks.
- Using the pi.dev interface.
Current Configuration:
- Context: Set to 64k (referencing ninfer-3090 builds, though llama-bench suggests 90k fits).
- Quantization: Using UD-Q4KXL, finding the XL variant sufficient compared to Q5.
- KV Cache: Experimenting with --cache-type-k q80 and --cache-type-v q40 to balance VRAM and quality, as 4-bit K significantly degrades performance.
Observations:
- pi.dev handles context limits well, managing auto-compaction delays gracefully.
- Current generation speed is slow; the user is seeking advice on better settings or whether to enable cache quantization.
More from Infra
- Academic lab lands 3M H100 hours plus £7M sponsored compute, a likely first — j_foerst · 2026-08-26
- Broadcom Reportedly Lends Money to OpenAI for Chip NRE, Solving Startup Funding Gap — MikePFrank · 2026-08-26
- VPS-Audit: Lightweight Open Source Script for Automated Server Security and Performance Auditing — tom_doerr · 2026-08-26
- Apple's Mini/Studio refresh and OpenAI's Jalapeño both put pressure on Nvidia — Stratechery · 2026-08-26
- Spain plans stricter rules for data centers on water, energy, and security — Polymarket · 2026-08-26
- TorchMorph: CUDA-Accelerated Morphological Transforms for PyTorch — kornia_foss · 2026-08-26