Run Qwen3.8 Locally: Unsloth Shrinks Model Size by 91% via 1-bit Quantization
danielhanchen · x · 2026-08-13
Unsloth AI announced that they have successfully shrunk the massive Qwen3.8-2.4T-A95B model from 4.9TB to 397GB (a 91% reduction) using their new Dynamic 1-bit quantization, enabling local execution.
- Quantization Breakthrough: By selectively quantizing layers and reducing codebook entries (down to 1.1875 bits per weight), the method retains accuracy while drastically cutting size. It works well for post-training quantization (PTQ) without needing QAT or QAD.
- Hardware Requirements: The compressed model can run on setups with 410GB+ RAM/VRAM via Unsloth Desktop or llama.cpp.
- Model Capabilities: Qwen3.8 features vision and thinking capabilities with a 256K context window (up to 1M tokens), rivaling GPT-5.6 Sol.
Related event: Unsloth Shrinks Qwen3.8 by 91% for Local Deployment(3 posts)→
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16
- Intel's neural-compressor: SOTA low-bit LLM quantization with FP4/NVFP4 support — HaihaoShen · 2026-10-16