Quantization evals have a noise floor: engine, restart and batch drift move up to 0.92% of decisions
Level5Tech · hf · 2026-11-18
The problem
Quantized models are usually judged against their BF16 originals, but that comparison also changes the serving engine, server instance, batching, and sometimes even kernel selection — all of which can flip decisions on their own.
Method
- Two Cloudflare decision models, Clef-flash (9B) and Clef (27B), measured on an NVIDIA GB10 over the same 2,400 public decisions
- Every quantization effect is stated as the excess over measured non-quantization noise sources
Key findings
- Non-quantization sources move 0–0.92% of decisions
- llama.cpp restarts are bit-identical; vLLM restarts flip 0/5/20 decisions (Clef-flash) and 4/0/22 (Clef) at BF16/FP8/NVFP4
- Switching engines at BF16 flips 6–16 decisions; batch-size changes 8–15; transformers silently selecting the Gated DeltaNet kernel moves 10–14
- Quantization within one engine moves more: 0.88%/1.04% (Q80), 2.12%/1.62% (FP8), 5.79%/3.50% (Q4KM), 5.79%/5.67% (NVFP4) for Clef-flash/Clef
- On both models every 4-bit interval excludes all non-quantization sources; two of five pre-registered predictions failed
Accuracy hides this
Over 31,275 questions, Clef's Q4KM shifts accuracy by just -0.02 points while 436 decisions individually get worse; NVFP4 costs 0.95 and 0.65 points. Code, quantization recipes, per-record outputs and pre-registration are released.
More from Infra
- Google donates Agent Substrate to CNCF: 500ms restores, 10x density — krishnan · 2026-11-18
- Report: Nvidia brokered Alibaba-Megaspeed deal, Alice Huang ties run deeper than known — GaryMarcus · 2026-11-18
- n8n's open-source agent Nod: secure remote access via private reverse proxy — victorialslocum · 2026-11-18
- pci_eerie: Open-Source PCIe Soft Stack for Xilinx 7-Series FPGAs as a Drop-in PCIE_2_1 Replacement — IgorCarron · 2026-11-18
- Hugging Face Nears 1 Exabyte of Monthly Data Distribution as Open Model Downloads Explode — andimarafioti · 2026-11-18
- NVIDIA, AMD, Intel, Google and Meta join SGLang Consortium as v1.0 ships — heyshrutimishra · 2026-11-18