Unlock 250k Context on RTX 4090: Optimizing Qwen 3.8 with Custom Drafters
alexcovo_eth · x · 2026-08-23
The author achieved a 250,000 token context window for Qwen 3.8-27B on a single RTX 4090 (24GB) running at 75 tokens/s through specific optimizations.
Key Optimization Techniques:
- Custom 2-bit Drafter: Quantized the DFlash 2 drafter to Q2K, saving 450MB of VRAM while maintaining 100% draft acceptance and decode speed (76 t/s) in benchmarks.
- Hidden VRAM Tax: Discovered that llama-server reserves massive chunks of VRAM for multi-user batching by default. By passing the --parallel 1 flag, the engine dedicates the full 24GB buffer to a single user session.
Results:
Combining the VRAM savings from the Q2K drafter with the --parallel 1 flag caused the context ceiling to "absolutely explode," enabling the massive 250k context length on consumer hardware.
More from Infra
- Llama-Mobile: 2.7-Bit Quantization Shrinks Llama 3.2 Vision 11B to 3.7GB for Phones — Luka Ribar · 2026-08-24
- DSCO Router Launches Unified Gateway for Multi-Model Routing with BYOK Support — arthurcolle · 2026-08-24
- Open Source RobotSoul: Persistent Identity for Agents After Context Resets — robauto-dot-ai · 2026-08-24
- Offloading MoE models to RAM causes slow prefill speeds — former_farmer · 2026-08-24
- Etched Raises $1B Led by Jane Street to Validate Architecture-Agnostic AI Chips — TheTuringPost · 2026-08-24
- ConvRot Quant joins llama-cpp: Q6 accuracy nears Q8 quality — giveen · 2026-08-24