SoloQ: Calibration-Free Quantization for Diffusion Language Models
Donghyun Lee, Arkapravo Ghosh, Varun Manjunath, Bumjoon Kyle Rhee, Hyunho Kook, Shiting Xiao, Youngeun Kim, Priyadarshini Panda
cs.LG
2026-10-06
SoloQ quantizes diffusion LLMs to W4A4 with no calibration data. On LLaDA-1.5 it scores 68.48 vs 69.23 FP16; NVFP4 cuts peak memory up to 2.61× and inference latency up to 2.24×.
A diffusion language model revisits bidirectional context at every denoising step. Full-sequence models such as LLaDA and Dream send the whole sequence through the network again each step. Block-diffusion models such as Fast-dLLM v2 and Nemotron-Labs-Diffusion write one block at a time and keep a KV cache for blocks already committed. Figure 1 profiles Fast-dLLM v2 on an H200: at short contexts the weights dominate memory; as the sequence grows, activations and the KV cache take a larger share, and attention gets expensive. Those costs repeat every step. Weight-only compression leaves them in place.
Activation scale moves too. The down-projection input shifts with the mask ratio and the denoising step. Existing dLLM post-training quantization (PTQ: compress a finished model, no retraining) fits that moving distribution with a calibration set. In Figure 1(c), QuaRot paired with GPTQ changes benchmark accuracy when the calibration set switches from WinoGrande to MMLU.
SoloQ comes from the Panda lab at the University of Southern California, with collaborators at Yale and Korea University. After a sufficiently random rotation, coordinates of a unit vector are approximately Gaussian with mean 0 and variance 1/d. A codebook can be fixed from that marginal, with no calibration set and no stored activation statistics. NVFP4 group scales are still read from the current tensor. What disappears is the fixed calibration sample.
RPBH, the block-Hadamard rotation OrbitQuant uses for image and video diffusion, is cheap, but energy that lands in the wrong block stays there. Dream-7B's feed-forward width is 18944 = 512×37, enough blocks for that leftover to show. K-RPBH adds a k×k random orthogonal mix after a random permutation and a per-block Hadamard, remixing the same coordinate across blocks. With k = 1 it is RPBH again. Proposition 1 says that if the mixer is Haar-random, the expected conditional variance of each coordinate is 1/d. On Nemotron activations, block-only rotation has KL 0.11396 to the target marginal and 67.89% block-energy imbalance. Adding the permutation and the cross-block mix brings those to 0.00413 and 22.12%.
Weights are rotated offline, activations online, and both are quantized in that shared basis.
Low bit-width also shortens vectors. A scale correction divides the original energy by the inner product of the original vector and its quantized version, which restores the projection along the original direction. Error orthogonal to that direction remains. The factor folds into the existing per-token or per-row scale, so the matmul bit-width does not change.
For block diffusion, KV entries are quantized only when a block is committed. The block still being denoised stays in full precision. Keys use KIVI-style per-channel asymmetric quantization. Values follow the activation path above.
W4A4 average accuracy (%) on full-sequence dLLMs. STaR-Quant is the strongest calibration-based baseline in every row.
| Model | FP16 | STaR-Quant | DLLMQuant++ | SoloQ-C | SoloQ-N |
| LLaDA-Base-8B | 58.95 | 57.07 | 54.29 | 57.74 | 57.60 |
| LLaDA-1.5-8B | 69.23 | 66.93 | 64.31 | 67.82 | 68.48 |
| Dream-7B | 66.54 | 63.59 | 61.90 | 62.06 | 62.53 |
On LLaDA-Base, SoloQ-C gains 3.45 points over DLLMQuant++. On LLaDA-1.5, SoloQ-N scores 68.48, 4.17 above DLLMQuant++ and 0.75 below the FP16 score of 69.23. GSM8K is where the gap opens. LLaDA-Base scores 69.52 in FP16, 67.78 with SoloQ-C, and 57.29 with STaR-Quant. LLaDA-1.5 scores 83.47, 81.80 with SoloQ-N, and 76.45 with STaR-Quant. Code runs the other way. On LLaDA-Base HumanEval, STaR-Quant scores 35.98, above the FP16 score of 31.71, while SoloQ-N falls to 25.61. On Dream-7B, STaR-Quant still leads the nine-task average.
Block-diffusion generation is brittle. Under W4A4, RTN and AWQ average 6.67 and 6.61 on Fast-dLLM v2, and 1.30 and 1.28 on Nemotron-8B, against FP16 averages of 60.35 and 78.67. QuaRot cannot cover Fast-dLLM v2's dimensions. On Nemotron, QuaRot+GPTQ still runs and averages 59.39; SoloQ-N scores 70.76 against an FP16 score of 78.67.
| Model | Setting | FP16 | SoloQ-C | SoloQ-N |
| Fast-dLLM v2-7B | W4A4 | 60.35 | 57.00 | 56.77 |
| Fast-dLLM v2-7B | W4A4KV4 | 60.35 | 56.19 | 56.21 |
| Nemotron-8B | W4A4 | 78.67 | 69.74 | 70.76 |
| Nemotron-8B | W4A4KV4 | 78.67 | 70.18 | 71.06 |
The Nemotron drop sits in instruction following. IFEval falls from 97.27 to 60.85 for SoloQ-N at W4A4, and to 64.11 with KV4 as well. GSM8K moves from 92.95 to 91.21 and MATH from 85.10 to 81.12. Adding 4-bit KV on Fast-dLLM v2 costs under a point on average.
With NVFP4, peak memory across the five dLLMs is compressed by 2.31× to 2.61× versus each model's own BF16 run, and end-to-end speed rises by 1.94× to 2.24×. SoloQ-N uses native NVFP4 kernels on an RTX Pro 6000. SoloQ-C uses 8-bit Tensor Cores on an H200. Nemotron's end-of-sequence position changes after quantization, so its 2.05× figure is measured on 75 forwards recorded from BF16.
Quantizing only the KV cache to about 3 bits per entry, with weights and activations left in FP16, SoloQ-N averages 53.53 on Fast-dLLM v2. KIVI averages 50.26, TurboQuant 4.03, and FP16 72.67.
The calibration set no longer has to stand in for every mask ratio and every denoising step. Weight rotation is offline. Online cost is an activation rotation plus a length correction absorbed into scales the kernel already applies. SoloQ-N is the path that matches NVFP4 hardware. The same rotation also tolerates plain uniform INT4: SoloQ-I averages 67.44 on LLaDA-1.5, against 68.48 for SoloQ-N, and 55.32 on Fast-dLLM v2, against 60.35 in FP16.
On full-sequence models the lead over STaR-Quant is often under two points. The practical difference is that no calibration set is required. On block diffusion, SoloQ is one of the few W4A4 setups in the main table that still finishes code and math generation. RTN and AWQ scoring near zero makes that lead look larger than the remaining gap to FP16.
Limitations are placed in Appendix G and are not listed in the main text. The tables already show where the result stops.
On Dream-7B, STaR-Quant's average of 63.59 beats SoloQ-N's 62.53. The abstract's claim of beating calibration-based methods on knowledge and reasoning tasks matches the LLaDA averages. Dream's overall average still favors STaR-Quant. Dream's TruthfulQA column is a weak comparison: DLLMQuant and STaR-Quant are not open-sourced, and the table note puts their FP16 baseline at 49.76 against 43.96 in this setup.
Folding the residual-stream rotation into neighboring weights removes some online rotation, and the equality stops holding after quantization. On Fast-dLLM v2, SoloQ-C falls from 57.00 to 50.94 and SoloQ-N from 56.77 to 52.81. On LLaDA-Base, SoloQ-N falls from 57.60 to 55.30. The rotation itself takes time: one activation rotation on Fast-dLLM v2 costs 5.779 ms with K-RPBH and 98.135 ms with a dense Haar rotation. On LLaDA-1.5 the same pair is 0.048 ms versus 0.559 ms.
The 2.61× memory compression and 2.24× speedup describe SoloQ-N against BF16. The codebook path runs on a different GPU, through 8-bit Tensor Cores. On a ZCU104 FPGA, NVFP4 reaches 0.252 tokens/s against 0.13 for BF16, which is 1.94× throughput and 2.70× energy efficiency on that prototype. Nemotron's block length and end-of-sequence policy are an operating point chosen for this paper, not an alignment with that model's original table.
At 4-bit KV, keeping the attention sink in full precision barely matters. At KV2 it does: Nemotron GSM8K scores 82.94 if the sink stays in full precision and 72.93 if it is quantized with the rest. The active denoising block stays in full precision, so long-context savings come from KV states that have already been committed.