Quantization evals have a noise floor: engine, restart and batch drift move up to 0.92% of decisions

Level5Tech · hf · 2026-11-18

The problem

Quantized models are usually judged against their BF16 originals, but that comparison also changes the serving engine, server instance, batching, and sometimes even kernel selection — all of which can flip decisions on their own.

Method

Key findings

Accuracy hides this

Over 31,275 questions, Clef's Q4KM shifts accuracy by just -0.02 points while 436 decisions individually get worse; NVFP4 costs 0.95 and 0.65 points. Code, quantization recipes, per-record outputs and pre-registration are released.

Original post →

More from Infra

Infra channel →