Unsloth Releases Qwen3.8 Local Deployment Guide with 1-bit Quantization Down to 397GB
MaziyarPanahi · x · 2026-08-13
Unsloth has released a local deployment guide for the Qwen3.8 model family. The series includes 27B, 2.4T-A95B (2.4T total/95B active params), and Max versions, featuring vision and thinking capabilities with up to a 1M token context window.
To handle the massive size, Unsloth extended llama.cpp's quantization with new 1-bit data types (e.g., UD-IQ1XXXS). By reducing codebook entries to 1.1875 bits per weight, the 2.4T model is compressed to 397GB (91% smaller) for local execution, retaining strong accuracy without requiring Quantization-Aware Training (QAT). The 27B model runs on 16GB+ memory setups.
Related event: Unsloth Shrinks Qwen3.8 by 91% for Local Deployment(3 posts)→
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16
- Intel's neural-compressor: SOTA low-bit LLM quantization with FP4/NVFP4 support — HaihaoShen · 2026-10-16