Quantizing a 9B model: 70% of damage sits in the first 8 layers, and perplexity hides the tail
DinnerPrestigious953 · reddit · 2026-10-23
Per-tensor measurements on a 9B reasoning model show 69.5% of quantization damage (KL vs the unquantized base) lives in layers 0-7 — exactly where standard imatrix allocators give the least protection. Perplexity reported the imatrix build as a clean 1.35-point win, yet its worst tensor was worse than the no-data heuristic (KLD 17.77 vs 17.65): PPL only sees averages, while agent failures (loops, stalls) live in the tails. A blind neural ranker (Gnom) trained only on damage measured from other models statistically tied the imatrix allocator on 164 HumanEval-style tasks and stayed closest to base outputs (mean drift -0.306%). Bit allocation even changed refusal rates: 14 vs 19 empty completions across builds. Dataset released on HuggingFace (wepiqx/MERNIK-Fracture).
More from Infra
- Power, not chips, may be AI's binding constraint as China's grid expansion looms — ingliguori · 2026-10-23
- Testing Intel Arc B60 for Local Inference, Planning a 12-Card Rig — glenbeer · 2026-10-23
- PHP WASM servers: 'The world is about to change a LOT' — MickeySteamboat · 2026-10-23
- DeepSeek V4.1 Flash test: official API hits 99% cache vs 92-96% on OpenCode Go — teortaxesTex · 2026-10-23
- a16z charts: GPU rental prices defy depreciation, up to 143% above trendline — a16z Newsletter · 2026-10-23
- The Jensen effect: MicroDucks sells 4,000+ units in 24h after GTC showcase — andimarafioti · 2026-10-23