KLD Is Misunderstood: Why It Can't Rank Quantized Models Reliably
Positive-Resource922 · reddit · 2026-10-25
A long Reddit post explains why the community overrates KLD as a quality metric for quantized models: it only measures average distribution divergence from the base model on a limited dataset. Random-text datasets don't reflect real inference distributions—better practice is scoring tokens the base model itself generates, with zero weight on prompts. KLD is also trivially cheap to game: evaluating on calibration data, or fine-tuning quantization scales on it, can make a small model beat twice-larger ones on KLD while generalizing poorly. Treat it as a diagnostic, not a leaderboard.
More from Models
- Reflection AI's open-weights Beam served 10 trillion tokens in one week — zacharynado · 2026-10-25
- Dimitris Papail: Raion 'incredibly strong for its size' as from-scratch pretraining spreads — DimitrisPapail · 2026-10-25
- Eric Jang: abliterated models show 'astounding' cyber capabilities — ericjang11 · 2026-10-25
- Community org claims to run "DeepSeek v4.1 flash" locally via pruning — unverified — amplifiedamp · 2026-10-25
- Kimi K3.1 disappoints: analysts say Moonshot trails DeepSeek on RL progress — scaling01 · 2026-10-25
- GPT-6.1 Sol beats Opus 5.5 on cost-performance balance, argues dev who caught Opus in logic traps — haider1 · 2026-10-25