ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization
He-Yen Hsieh, H. T. Kung
cs.LG, cs.CL
2026-08-11
ReRound uses a weight diffusion prior to flip under 1% of rounding decisions, beating RTN at 3- and 4-bit and surpassing GPTQ and SignRound, which need calibration data.
When you compress model weights to 3 or 4 bits, the default move is round-to-nearest (RTN): snap each weight to the closest quantization grid point. The trouble is weights that land almost exactly between two grid points. Rounding up or rounding down costs nearly identical error on that single number, so RTN's pick at those positions is effectively a coin flip. This is "midpoint ambiguity."
It hurts small models disproportionately. A 70B model has enough parameters to average out a few bad rounding calls; a 1B model does not, and the errors accumulate into visible accuracy loss. Small models are also the ones that most need aggressive quantization, since they ship to phones and edge devices where every bit counts.
ReRound's move is simple: if a single weight cannot tell you which way to round, ask the model what that weight was more likely to be. Three steps.
First, train a conditional diffusion model to reconstruct continuous weights. It reuses the DeepFloyd IF Stage-II U-Net with a ControlNet conditioned on the low-bit weight patch. The authors slice every weight matrix into 64x64 patches, pair each full-precision patch with a stochastic-rounding low-bit version during training, and let the diffusion model learn to recover continuous weights from coarse observations. A clean design choice: weight recovery needs no language information, so the text encoder gets a precomputed embedding of an empty prompt.
Second, let that reconstruction change its mind, but only near midpoints. At inference, deterministic 2-bit RTN patches serve as the condition, and the diffusion model produces a reconstructed matrix Wrec. ReRound flips a rounding decision only when the reconstruction points the opposite way and the deviation stays within a position-dependent tolerance, which is large near midpoints and tight near grid boundaries. Net effect: ReRound revises at most 1% of RTN assignments in any matrix, and the overwhelming majority stay as RTN decided.
Third, pick the best candidate. ReRound sweeps the tolerance parameter to generate several candidate quantized matrices and keeps the one whose top-k singular values best match the full-precision weights. Singular values measure global structural fidelity rather than element-wise error.
The crucial property: ReRound reuses RTN's scales, zero-points, and group size, changing only rounding direction. The quantized representation and inference path are identical to RTN, so there is zero added cost at deployment. All the cost is offline.
Against other calibration-free methods, on five models (Gemma 2 2B, Gemma 3 1B, Qwen3 1.7B, OLMo 2 1B, SmolLM2 1.7B), ReRound beats its matched group-wise RTN by 0.2 to 1.3 points at 4 bits and 0.1 to 1.6 points at 3 bits, on every model, with no regressions. At 3 bits it has the best four-task average (WinoGrande, PIQA, BoolQ, SIQA) on all five.
| Model | Config | RTN (group) | ReRound | Comparison |
| Gemma 2 2B | W4A16 | 66.4 | 67.4 | SignRound 67.0 |
| Gemma 3 1B | W4A16 | 58.8 | 59.3 | SignRound 59.1 |
| OLMo 2 1B | W4A16 | 58.1 | 59.4 | HQQ 60.4 / BNB FP4 61.1 |
| OLMo 2 1B | W3A16 | 54.3 | 55.1 | HQQ 54.4 |
More striking: with no calibration data, it beats methods that need it. On Gemma 2 2B, GPTQ scores 66.7, AdaRound 66.9, SignRound 67.0, and ReRound 67.4 tops all three. Gemma 3 1B is the same, ReRound 59.3 over SignRound 59.1.
One honest caveat. On OLMo 2 1B, ReRound (59.4) is actually below HQQ (60.4) and BNB FP4 (61.1). Those use different quantization parameterizations, with different scales and grouping. Held to the fixed RTN parameters, ReRound still lifts RTN from 58.1 to 59.4, the largest 4-bit gain in the set.
It also stacks on top of other PTQ methods. Re-running with SINQ's scales, 4-bit four-task average rises from 62.1 to 62.6 and average perplexity drops from 18.53 to 18.46. ReRound optimizes rounding given any parameter set, not the parameters themselves.
For on-device and edge deployment, calibration data is a hidden cost. You need representative inputs, with all the privacy and distribution headaches that brings. ReRound reads only the model's own weights, needs no activations or text, and deploys exactly like an RTN model with no inference overhead. That is a real convenience for the common case of taking an open model and squeezing it onto a device.
It also opens a lane distinct from tuning quantization parameters. GPTQ, AWQ, and HQQ mainly work on scales, grouping, and transforms. ReRound shows that revisiting rounding alone, with parameters fixed, gains reliably and can be layered onto other PTQ methods.
The cost is concentrated offline. Each target model needs its own diffusion model: 1.76 to 3.43 hours to train (2 GPUs) and 3.65 to 9.55 hours to reconstruct (1 GPU). The authors reuse one reconstruction across 3- and 4-bit runs to save some of this, but scaling to many models stays expensive.
The candidate set is bounded by external parameters. ReRound inherits scales and grouping from another PTQ method and can only revise rounding within that frame. Joint optimization of rounding and parameters is left as future work.
The selection criterion is weight-space only. Matching singular values measures structural fidelity and need not pick the best candidate for every downstream task. Patch-based reconstruction (64x64) may also miss long-range and cross-layer dependencies.
Finally, the paper validates only 1B to 3B models. Whether gains hold at 7B or 70B, where midpoint ambiguity may wash out, is unanswered. The authors claim applicability beyond LLMs but offer no experiments.