Pruned-vocab Qwen3.8-27B fits 75K context on 16GB at ~60 tok/s, IFEval unchanged
fakezeta · reddit · 2026-11-15
The author condensed Unsloth's Qwen3.8-27B Q3KXL with bsaleh03's ASCII-Condensed prune tool (keeping only ASCII + Latin-1/Latin-Extended-A), built GGUF and ninfer versions on Hugging Face, adapted iamwavecut's conversion scripts for the smaller vocabulary, and opened a PR upstream. Quality holds: IFEval 84.4% vs 83.9% base (within std dev). On a 5060 Ti 16GB with rk8v4 KV cache it reaches 75K context at 60 tok/s across 541 IFEval samples. A lighter version using Unsloth's separately released low-quant MTP is planned to save 90MB more RAM.
More from Infra
- Running Qwen3.8-27B on a 12GB RTX 3060 with 65K context: full recipe inside — WiseVanilla2743 · 2026-11-15
- Cisco hits $9B AI revenue, beats and raises, yet shares fall 6% anyway — pdamodaran · 2026-11-15
- AWS investigates connectivity issues affecting multiple services in us-east-1 — kingslyj · 2026-11-15
- RustingStrata 1.2.3 ships a built-in local chat page in a single Rust binary — Serious_Scarcity7818 · 2026-11-15
- Congress reportedly prepping tighter AI chip export controls with bipartisan support — Polymarket · 2026-11-15
- ggml tensor library trends on GitHub with 15.8k stars, up 154 in a day — ggml-org · 2026-11-15