EdgeRazor (Nanjing LAMDA + Microsoft AI): 1.88-bit LLMs beat 2-bit and 3-bit baselines

EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng, Xin-Yi Zou, Chen Wu, Nan Li, Shao-Qun Zhang, Zhi-Hua Zhou

cs.LG, cs.AI

2026-04-10

EdgeRazor (Nanjing LAMDA + Microsoft AI) distills Qwen3-0.6B to 1.88-bit mixed-precision, beating every 2-bit and 3-bit baseline across 14 tasks; the 1.58-bit build decodes 15x faster.

What problem this solves

To run large language models on phones and other constrained devices, sub-4-bit quantization is almost unavoidable, and it is exactly where current methods fall apart. Post-training quantization (PTQ) stays near-lossless at 4-bit but degrades badly at 2-bit and 1.58-bit. Quantization-aware training (QAT) recovers accuracy but needs heavy compute and large corpora to retrain. The middle ground, quantization-aware distillation (QAD), is held back by three old problems: bit allocation is heuristic, the layers used for distillation are picked by hand, and the loss assumes a single data distribution. EdgeRazor targets all three.

Method

EdgeRazor is short for Mixed-Precision Quantization-Aware Distillation (MPQAD), with one module per problem.

Structural Quantization with Mixed Precision (SQMP) handles bit allocation. It packs every ⌊1/ρ⌉ consecutive output channels into a super-group, gives one channel 4-bit and the rest 1.58-bit, with ρ controlling the 4-bit fraction (ρ=1/8 gives 1.88-bit, ρ=0 gives 1.58-bit), instead of scoring each channel for sensitivity and handing more bits to the sensitive ones. The reason: during QAT/QAD the weights keep updating, so the channels that looked important at initialization drift away by mid-training, and PTQ's static sensitivity assignment breaks under training. A fixed periodic pattern sidesteps the need to chase drifting importance.

Layer-Adaptive Feature Distillation (LAFD) handles which layers to align. It computes the mean cosine similarity between adjacent teacher layers and picks the k layers with the lowest similarity, the ones that change the representation direction the most. The appendix supplies the key evidence: which layers transform representations most is strongly domain-dependent, with math and code corpora lighting up different layers. Fixed "always supervise layer N" heuristics are wrong from the start, so reading the importance off the teacher itself beats hand-tuning.

Entropy-Aware KL Divergence (EAKLD) handles the loss. Logit distillation has to balance forward KL (cover all of the teacher's modes) against reverse KL (chase only the teacher's strongest mode). The older CAKLD switches between them with a fixed rule, which assumes a stable data distribution; real corpora often mix human-annotated and externally distilled data with different entropy profiles, so a fixed rule mismatches. EAKLD uses the teacher's output entropy as the switch: high entropy (the teacher is unsure) pushes toward forward KL, low entropy (the teacher is confident) pushes toward reverse KL.

The shared move across all three is to replace hand-set or drift-prone judgments with quantities computed from the data or the teacher itself.

Results

The main testbed is Qwen3-0.6B averaged over 14 domain tasks, under the harder weight+activation regime:

Setting	EdgeRazor	Strongest baseline	Gap
1.88-bit	41.76	2-bit OmniQuant 30.49	+11.27
1.88-bit	41.76	3-bit FlatQuant 37.38	+4.38
4-bit	47.80	FlatQuant 45.74	+2.06

The 1.88-bit EdgeRazor beats every 2-bit and 3-bit baseline at once. Under weight-only quantization, 4-bit EdgeRazor averages 47.83, 0.48 above the BF16 original, essentially lossless. On the quantization-sensitive GSM8K (math) and HumanEval (code), most 2-bit baselines collapse toward zero while EdgeRazor holds up.

It generalizes. On MobileLLM-350M, EdgeRazor beats QAT-finetuned ParetoQ at every bit-width while consuming only a quarter to a tenth of the training tokens. On the multimodal Qwen2.5-Omni-7B at 4-bit, EdgeRazor (with an extra 4-bit vision encoder) beats AWQ by 0.44 on Video-MME and 1.42 on MLVU.

Deployment efficiency (1.58-bit Qwen3-0.6B, Apple M4 Pro, llama.cpp): storage drops from 1.11 GB to 0.19 GB, memory from 1.46 GB to 0.51 GB, prefill speeds up 2.11x, decoding 15.16x (20.91 to 317.03 tokens/s). With groupsize 256 the quantization proportion reaches 99.99%, embedding and lmhead included.

Why it matters

For anyone running LLMs on the edge, this is direct evidence that sub-2-bit is finally usable. Sub-4-bit used to mean a broken model; EdgeRazor pushes the usable floor down to 1.88-bit and even 1.58-bit, and it does so without piling on training, in fact using far less than mainstream QAT. A 0.6B model in 0.19 GB with 15x faster decoding brings small models that genuinely fit phone-class memory one step closer.

Honest caveat: this is incremental engineering integration. None of the three modules (mixed precision, adaptive distillation, entropy-adaptive KL) is a brand-new idea on its own. The contribution is redesigning each against QAD's specific failure modes and assembling them into a framework that stays stable at ultra-low bit-widths. The breakthrough is not a single point; it is that this path now works end to end.

Limitations

No dedicated Limitations section. The claim of turning aggressive quantization from a fragile trade-off into a resilient approach rests on 14 multiple-choice-style downstream tasks (commonsense, MMLU, GSM8K, HumanEval, etc.), with no perplexity or open-ended generation quality reported.
It still requires training. EdgeRazor is not zero-shot PTQ; each model trains for 1 to 5 epochs on Infinity Instruct, a distilled reasoning set, and math/code corpora. Cheaper than ParetoQ, but not free.
Hyperparameters are model-sensitive. ρ, the entropy bound K, the layer count k, groupsize (256 for Qwen3, 64 for MobileLLM, 32 for Qwen2.5-Omni), and the loss weights all need per-model tuning; there is no single universal knob.
Efficiency numbers come from llama.cpp on an Apple M4 Pro and an i9-14900K (CPU), with no real phone-NPU or edge-ASIC results, so on-device gains need separate validation.
"1.88-bit beats 2/3-bit baselines" only reads fairly within the same quantization regime: it holds for weight+activation, but EdgeRazor's 1.88-bit is an average mixed from 4-bit and 1.58-bit, not the same kind of object as a uniform "2-bit" baseline, so comparing raw bit counts is apples to oranges.
The bit floor stops at 1.58-bit; it does not reach the true 1-bit regime of BitNet, so there is headroom left.

Terms

PTQ (post-training quantization): calibrate a finished model on a small set without retraining; near-lossless at 4-bit and above, degrades sharply below.
QAT / QAD (quantization-aware training / distillation): simulate quantization error during training, usually with a full-precision teacher for distillation; stronger than PTQ below 4-bit but requires training.
Forward / reverse KL divergence: two ways to measure the gap between student and teacher output distributions. Forward KL makes the student cover all of the teacher's modes; reverse KL makes it chase the teacher's strongest mode.
super-group: the unit of EdgeRazor's mixed precision, a block of consecutive output channels with one channel at 4-bit and the rest at 1.58-bit.
salience drift: the ranking of weight importance shifts during training, so channels flagged as sensitive at initialization may not stay sensitive mid-training.