Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification
Hong Huang, Decheng Wu, Qiangqiang Hu, Guanghua Yu, Jinhai Yang, Jianchen Zhu, Xue Liu, Dapeng Wu
cs.LG, cs.AI
2026-01-12
Sherry packs ternary weights at 1.25 bits via 3:4 sparsity. LLaMA-3.2-1B ties Tequila at 0.519 average; a 0.7B model hits 148.27 tok/s on an i7, smaller and faster than 2-bit.
Ternary quantization pins weights to {-1, 0, +1} and turns multiplies into table lookups plus adds, which is attractive on the edge. Packing those values onto a CPU still forces a bad choice. Pad each weight to 2 bits and you have wasted the extra state versus INT2. Pack three weights into 5 bits (about 1.67-bit) and the 3-way groups miss power-of-two SIMD lanes, often running slower than 2-bit. Theory sits near 1.58 bits. Practice either wastes bits or wastes throughput.
Sherry's move is to turn the zeros that ternary already has into regular zeros.
The constraint is strict: in every four consecutive weights, exactly three are nonzero (±1) and one is zero. C(4,3) times 8 sign patterns is 32 states, which fills 5 bits, 1.25 bits on average. Block size 4 matches SIMD. Sparsity is 25%; the paper cites prior work that ternary quality collapses past 50% zeros. The 5-bit index splits into a sign bit plus 4 lookup bits, which fits the 16-byte table of AVX2 vpshufb. Quantization is greedy Sparse-AbsMean: zero the smallest absolute value in the block, take signs on the rest, set the channel scale to the mean absolute value of the kept weights.
Hard 3:4 pruning polarizes the weights. Gradient effective rank falls into binary territory (ER < 750 on a 4096-wide matrix). The paper calls this weight trapping. Arenas adds an annealing full-precision residual during training: Y = XT α + λt XW, with λt cosine-decayed to 0. That residual hooks latent magnitudes back to the loss so gradients are not flattened by a uniform ±1/0 pattern. At inference the residual is gone. QAT uses 10B UltraFineWeb tokens, group size 128 by default, on LLaMA-3.2 1B and 3B.
Inference rides BitNet.cpp: pack each 4-tuple into index plus sign offline, build local LUTs from activations online, accumulate, then apply the channel scale.
On LLaMA-3.2-1B, five-task average is 0.519 for Sherry and 0.519 for 1.67-bit Tequila, against 0.558 BF16. ARC-c is 0.309 versus Tequila 0.305 and BF16 0.313. On 3B, Sherry is 0.567, Tequila 0.576, Spectra 3.9B 1.67-bit 0.567. Versus BitNet 1.3B at 0.483, the 1.25-bit 1B model is about 3.6 points higher at a quarter fewer bits.
| Model | Bits | 5-task avg | Reference |
| Sherry 1B | 1.25 | 0.519 | Tequila 0.519 / BF16 0.558 |
| Sherry 3B | 1.25 | 0.567 | Tequila 0.576 / BF16 0.636 |
| 0.7B CPU | 1.25 | 148.27 tok/s, 205.5MB | TL2 116.83 / 233.4MB |
| 3B CPU | 1.25 | 45.55 tok/s, 712.4MB | TL2 38.80 / 846.0MB |
Granularity: per-tensor 0.502, per-channel 0.513, group-128 0.519. Arenas keeps coarse grids usable. Ablations show gains on 1-bit, 1.25-bit, and 1.67-bit; linear, exponential, and cosine decay all beat the no-Arenas run, warmup helps. The abstract's "zero accuracy loss, 25% fewer bits, 10% faster" matches the 1B average versus Tequila. The 3B average is 0.009 lower. Speed on 3B is about 18% above 1.67-bit and about 9% above 2-bit, so "10%" is a blurry headline. Size versus 1.67-bit on 3B is 712MB from 846MB, about 16%.
The ternary literature has been stuck between 1.58/1.67 and padded 2-bit. Sherry shows structured sparsity can buy a hardware-friendly 1.25-bit width without falling out of Tequila's accuracy band. The sweet spot is CPU LUT engines and 1-3B models. Arenas as a train-only bypass is reusable for other hard quantization constraints, with zero inference cost.
This is not a 70B server recipe and not activation quantization.
The authors list four: scale stops at 3B; no NVIDIA Sparse Tensor Core work; weights only, activations and KV stay BF16; Arenas adds QAT compute. A few more. The suite is PIQA, ARC, HellaSwag, WinoGrande; no MMLU, GSM8K, or long context. Reproduced QAT baselines share the 10B token recipe, while BitNet and Spectra numbers come from the original papers, so data is not aligned. "Zero loss" holds for 1B versus Tequila; 3B already gives up the average. Code lives in AngelSlim. Reproducing still means 32-GPU QAT.