Serving many fine-tuned variants of a 27B model on one GPU: 4-bit CQ format and 55x smaller adapters
textclf · reddit · 2026-11-14
The thousandml team open-sourced two methods for multi-tenant fine-tuned model serving on Qwen3.8-27B, with reproducible benchmarks.
CQ 4-bit base format
- Calibration-free 4-bit format; the 27B model shrinks from 55.6 GB (BF16) to 18 GB
- No significant difference vs BF16 across 9 benchmarks, while NF4 (bitsandbytes) loses 12 points on GSM8K (p=3e-6) — meaning most QLoRA setups start from a weaker base
GCMP, a LoRA replacement
- Only 2.9M trainable params (11 MB) vs 159M (320 MB) for LoRA r=32
- Statistically indistinguishable from LoRA on 9 benchmarks, slightly lower validation loss
- 1,000 adapters take 11 GB instead of 320 GB
Deployment and limits
- vLLM serves many GCMP adapters from one CQ base, chosen per request and batched together; works on full-precision bases too
- Limitations: 2x LoRA training step time; vLLM serving currently requires a CQ base; pipeline parallelism works, tensor parallelism doesn't yet; validated on one model/dataset so far
More from Infra
- Perplexity team teases minimalist LLM trainer built with coding agents, open source soon — BanghuaZ · 2026-11-14
- Musk: Future Starship Flights Will Carry Over 50 Tb/s of Starlink Bandwidth to Orbit — elonmusk · 2026-11-14
- Fractile CEO Walter Goodwin on challenging Nvidia, $720M Series C at $7.5B valuation — 20VC · 2026-11-14
- Chart resurfaces showing GitHub outages spiked after Microsoft acquisition — zacharynado · 2026-11-14
- Targon SDK update unifies VMs, bare metal and sandboxes in Python, Go and TypeScript — JosephJacks_ · 2026-11-14
- AWS Emulator Goes GA: 200+ Services Run Locally, Built Almost Entirely by AI Agents — iamfakhrealam · 2026-11-14