The Failure Zones of Quantitative Evaluation Metrics
enrique-byteshape · reddit · 2026-07-16
This post summarizes an in-depth analysis of quantitative model evaluation: KLD and perplexity help rank models when "degradation is already obvious," but in low-loss regions near the baseline, they are almost useless for determining which quantized model is actually better.
Key conclusions include:
- The study compared 28 quantized models, analyzing KLD rankings, quality rankings, as well as BPW rankings and actual tokens/s throughput.
- The author identified a silent zone: below a certain KLD threshold, KLD has almost no correlation with actual quality. Lowering KLD does not mean better quality.
- In this zone, KLD still reflects the "amount of difference from the BF16 reference" but cannot tell if those differences are beneficial or harmful.
- Only when the model enters a more obvious lossy zone does divergence more reliably correspond to quality degradation.
- BPW has similar issues: it helps estimate model size and whether it fits on a device, but cannot reliably rank the actual throughput of same-size quantized models.
The post ends with links to the blog series and preprint, noting that full results are now public.
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27