Qwen3.8-27B Quantization Analysis: 4-bit Damage Peaks at Context Start
sadnessdevil · reddit · 2026-08-23
This post evaluates the KL Divergence (KLD) of various Qwen3.8-27B quantization checkpoints using code datasets on 8k and 32k contexts.
Key Findings:
- Huge Gap Between 8-bit and 4-bit: The best 4-bit score (0.00835) significantly outperforms the worst 8-bit (0.00071).
- High Variance in 4-bit: Models with identical sizes show KLD ranging from 0.01364 to 0.02976, indicating file size is a poor metric for 4-bit selection due to calibration/rounding differences.
- Group Size Impact: Checkpoints with group size 32 outperformed those with group size 128, even when the latter kept some modules in BF16.
- Damage Concentrated at Context Start: Surprisingly, all 24 models show much higher KL divergence in the first 500 tokens.
- Specific Layers Safe: Quantizing lmhead, embedtokens, and linearattn showed no performance impact in this test.
More from Research
- Paper proposes method for Jensen-Shannon centroid of categorical distributions — FrnkNlsn · 2026-08-23
- Building a text watermark remover for Claude, rewriting works best — Imaginary_Dinner2710 · 2026-08-23
- RSI replaces scaling as core AI strategy in 2026; AIBuildAI emerges — 机器之心 · 2026-08-23
- PlayWorld: Benchmarking world models via long-horizon agent gameplay — 机器之心 · 2026-08-23
- MIT Professor on AI for Science: The age of abundant discovery is here — ProfBuehlerMIT · 2026-08-23
- Ox Alpha: The troll LLM architecture that updates latent states — iruletheworldmo · 2026-08-23