ThousandML Open-Sources CQ 4-bit QLoRA: 17.4% Lower Val Loss Than Unsloth NF4
textclf · reddit · 2026-10-26
The author launched ThousandML, a fine-tuning platform built around their own calibration-free 4-bit quantization CQ (Coded Quantization). On Qwen3.8-27B, CQ 4-bit beats unsloth-bnb-4bit on Wikitext-2 KLD metrics (mean 0.0282 vs 0.0330) at smaller disk size (18GB vs 22.4GB). A QLoRA benchmark on Alpaca (single A40 48GB, 500 steps, r=32) showed 17.4% lower final validation loss (1.0315 vs 1.2484) and apparently faster convergence—but steady-state step time is far slower (27.6s vs 3.2s), which the author is working to fix. The post includes a full quickstart: install, download the CQ 4-bit base, train a LoRA adapter, and serve it with vLLM (--quantization cq, --enable-lora). Caveats: loss-only comparison, downstream quality and tokenization/loss-mask equivalence not independently verified.
More from Infra
- gpu-unlock boosts RTX 3090 Ti BF16 GEMM 87.5→157.4 TFLOPS — jiqizhixin · 2026-10-26
- Anthropic engineer: Claude Code cache hit rates are now high, savings live — teortaxesTex · 2026-10-26
- Six-month bet: hybrid AI deployments, not a wholesale shift to local models — yi111 · 2026-10-26
- Deco decouples 'thinking' from 'spelling' in generative recommenders for 3.2-8.6x throughput — _reachsumit · 2026-10-26
- Redditor with $4,000-$5,000 budget seeks local AI video generation hardware advice — dewsphere · 2026-10-26
- Why OpenAI and Anthropic Cut LLM Prices by Up to 80% in Months — Azer_pubes33 · 2026-10-26