CQ 4-bit matches BF16 while NF4 drops 12 points on GSM8K, plus GCMP with 55x fewer params
textclf · reddit · 2026-11-14
The author reran benchmarks of their calibration-free 4-bit format CQ against bitsandbytes NF4 on Qwen3.8-27B with unified lm-evaluation-harness settings, greedy decoding, and paired statistical tests (McNemar + bootstrap), addressing earlier criticism that cross-trainer loss comparisons were invalid.
Key findings
- CQ 4-bit (18 GB) shows no significant difference from BF16 on any of 9 tasks; NF4 (22.4 GB) loses 12.0 points on GSM8K (p=3e-6). CQ beats NF4 by 8.7 points on GSM8K (p=0.0005).
- GCMP, a LoRA replacement, trains only 2.9M parameters (11 MB) vs 159M for LoRA r=32 (320 MB), with no significant benchmark differences and slightly lower answer/validation loss. Both adapters add 21 points on GSM8K and cost 2.5 on TruthfulQA.
- GCMP currently takes 2x LoRA's step time; Unsloth NF4 remains fastest. Kernel optimization is ongoing.
- Serving: vLLM runs CQ via --quantization cq; GCMP adapters can be swapped per request. Pipeline parallel works, tensor parallel doesn't yet.
Caveats: one model, one dataset (Alpaca), quick subsets of 200–1,140 questions detect 5-point differences only. The author admits the earlier 0.84 vs 1.25 loss comparison was invalid. Code is open source with exact reproduction commands; CUDA kernels ship compiled with a free early-access license check.
More from Infra
- Nvidia's supply chain clout tested after Jensen Huang, whose financing backed neoclouds — pstAsiatech · 2026-11-14
- Amodei: compute is Anthropic's 'deepest business dilemma' ahead of IPO — SuB8u · 2026-11-14
- Anthropic IPO roadshow slide leak sparks Blackwell training speculation — and a block — teortaxesTex · 2026-11-14
- New Yorker interview: AI is a brute-force problem that will 'cook the planet' — kevinnbass · 2026-11-14
- Lenore unveils hybrid Token factory in Zhangjiakou: 15,000P cluster, 2 quadrillion tokens a year — 量子位 · 2026-11-14
- LexiSuite: a Rust-built inference engine beats llama.cpp 1.0.0 at 262k context — W61k3r · 2026-11-14