Low-bit quantization findings: PTQ works at 4 bits, QAT matters below that
Mindless_Ad_407 · reddit · 2026-09-13
The author shares takeaways from low-bit quantization experiments, releasing the QUASAR QAT paper and checkpoints.
- At 4 bits: good PTQ gets close. On Qwen3.5-4B (same 200/200 projections quantized), KL divergence: QUASAR QAT 0.0181, GPTQ 0.0227, standard QAT 0.0228, RTN 0.0423.
- At 3–2 bits: PTQ becomes brittle and QAT with quantization error in the training loop pulls ahead; at 2 bits QUASAR QAT beats the strongest baselines by 3–4 points on average across 8 tasks.
- Method: reconstruct quantized weights in a loss-aware way during training instead of simple rounding; no special inference path needed.
Paper and checkpoints are on arXiv and Hugging Face; the author invites others' experience at 2–3 bits.
More from Research
- MIT team reports in Science how 3D folding of synthetic gene circuits shapes expression — anshulkundaje · 2026-09-13
- Upstream gene dominance: supercoiling-mediated repression gates downstream activation — anshulkundaje · 2026-09-13
- OpenAI open problem #2 cracked by human theorists; paper unifies and improves MRRW bounds — abeirami · 2026-09-13
- Redditor claims GPT-6 solved 3D two-phase conductivity-function closure, unverified — Severe-Ad8673 · 2026-09-13
- Dead fly's brain booted to read letters: 166,700 neurons simulated live — alvelda · 2026-09-13
- Patched vLLM+FlashInfer Pushes Gemma 4 31B to 150 tok/s on a Single B300, Beating SGLang — abhijithneil · 2026-09-13