Bits or permutations? Intel prices quantization vs option-order averaging for small local LLMs
Mohammed Nayeem Khan · hf · 2026-11-12
An Intel study asks how a fixed local-LLM inference budget should be split between weight precision and averaging over answer-option reorderings.
Setup: Qwen3-0.6B/1.7B with simulated round-to-nearest quantization (8/6/4/3 bits, fp16 reference), reading A–D logits on ARC-Easy, ARC-Challenge, MMLU and HellaSwag (300 items each).
Findings:
- 6 bits lose no accuracy vs fp16 (scoring +1.8 and +0.9 higher, unexplained)
- At 4 bits, answer-slot skew rises from 0.15 to 0.53 (0.6B) and 0.07 to 0.12 (1.7B)
- Averaging four cyclic orders recovers +12.6 points at 4 bits for 0.6B
- Pricing bits and passes in one unit (weight bytes read × passes), extra orders only beat extra bits when weights have ≥6 bits; the decision rule: use 6-bit weights and spend the rest on up to four orders, one pass each
Caveat: simulated, two small models, simplest quantizer; no earlier study of quantization plus option order was found.
More from Research
- Microsoft's AI-assisted scan surfaces 59 Linux kernel bugs, 33 CVEs, fixes merged — mark_k · 2026-11-12
- CreativityPrism benchmark scores 17 frontier and open LLMs across 8 creative tasks — DanielKhashabi · 2026-11-12
- Possible eval leak: all frontier models fail a GPQA Diamond question whose official answer seems wrong — qberthet · 2026-11-12
- Latent Sparse Attention clusters KV values for 70x speedup over full attention — bendee983 · 2026-11-12
- Flash ALOHA masters precise robot manipulation with just 50 demos, fully open sourced — Thom_Wolf · 2026-11-12
- CARTBench from Zhejiang U & Xiaohongshu: no model tops 45% on excellent shopping advice — Yuqin Dai · 2026-11-12