Disaggregated Quantization: Specializing LLM Prefill and Decode
Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh
cs.LG
2026-09-22
DQ trains an NVFP4 prefiller for frozen 1-bit GGUF: +32.5 MMLU-Pro and +35.3 MMMU-Pro on Qwen3.8-27B, 1.78x faster 8K TTFT in llama.cpp.
Prefill and decode pull quantization in opposite directions. Once the prompt is long, each weight is reused across many tokens, so linear layers are compute-bound and native low-precision GEMM such as NVFP4 or INT4 helps. At batch one, decode does a single multiply-add per weight; runtime is dominated by fetching those weights, so 2-3-bit weight-only encodings win.
Most stacks still use one format for both. Weight-only kernels miss hardware GEMM on prefill. W4A4 speeds prefill but also quantizes activations on decode, which hurts generation more. On Qwen 3 and Gemma 3, decode-only NVFP4 costs 2-4x the accuracy loss of prefill-only NVFP4 on decode-heavy tasks, and 7x on Gemma3-1B. Prefill-heavy tasks reverse that ranking.
Disaggregated quantization (DQ) keeps the pretrained attention layout and specializes linear layers by phase: compute format, weights, and where those weights live. Quantization-aware distillation with disaggregation (QADD) reuses the SFT assistant mask as a pathway switch. User tokens run prefill; assistant tokens run decode. The loss is KL(teacher ∥ student) on response tokens only. Gradients still reach prefill weights through the prompt keys and values that decode attends to, so one forward-backward pass trains both paths. The run uses 100M non-padding Tülu 3 tokens against a frozen BF16 teacher.
Released GGUF decoders can stay frozen. Only the NVFP4 prefiller is trained, so it learns the KV that decoder actually consumes. The quantization recipe can remain opaque; the checkpoint is enough.
Core QADD covers Qwen 3 at 0.6B, 1.7B, 4B, 8B and Gemma 3 at 1B, 4B, 12B. Decode-heavy scores average GSM8K, MATH-500, and MMLU-Pro (Qwen 3 with and without think). Prefill-heavy scores average 13 RULER tasks at 4K, 8K, 16K, 32K. Family numbers are unweighted means over sizes, using the last five QADD checkpoints. Prefill latency is the transformer stack at 16K on DGX Spark; decode is vLLM batch-one per-token latency.
| Setup (Qwen 3 family) | decode-heavy | prefill-heavy | 8B prefill | Device weights |
| BF16 | 66.2 | 84.2 | 1.00× | 16.38 GB |
| Uniform NVFP4 | 61.8 | 79.8 | 1.49× | 6.40 GB |
| Format-disagg. NVFP4 | 63.7 | 81.1 | 1.49× | 6.40 GB |
| Uniform LUT2 | 34.8 | 61.3 | 1.49× | 4.66 GB |
| Fully disagg. LUT2 | 45.5 | 76.6 | 1.49× | 8.57 GB |
| LUT2 + ODP | 45.5 | 76.6 | 1.47× | 4.66 GB |
Relative to uniform quantization, format disaggregation adds 1.9/3.1 points on decode-heavy NVFP4 (Qwen/Gemma), 4.5/3.7 on LUT3, and 2.5/1.8 on LUT2, and under 1.3 points on prefill-heavy tasks. Weight storage and prefill speed stay put. Decode is 2-3% faster because activation quantization is skipped.
Full disaggregation moves the 2-bit needle. On Qwen 3, LUT2 decode-heavy goes from 34.8 to 45.5 and prefill-heavy from 61.3 to 76.6 versus the uniform LUT2 path. Versus weight-only LUT2A16 the lift is 4.5-12.5 points, with faster prefill. On Gemma 3, fully disaggregated LUT2 takes decode-heavy from 30.8 to 38.2 and prefill-heavy from 50.8 to 61.3. ODP brings Qwen3-8B device weights back from 8.57 GB to 4.66 GB, adding under 5% prefill latency above 16K. On Gemma3-12B, ODP at 16K is still 1.58× BF16.
An NVFP4 prefiller on frozen Unsloth GGUF checkpoints of Qwen3.8-27B, decode weights untouched:
| Decode format | MMLU-Pro | MMMU-Pro |
| IQ1S wo / +prefiller | 29.04 / 61.54 (+32.50) | 24.39 / 59.65 (+35.26) |
| IQ2XXS | 70.50 / 77.93 (+7.42) | 59.60 / 65.90 (+6.30) |
| IQ3S | 83.65 / 83.02 (-0.63) | 74.74 / 71.97 (-2.77) |
Accuracy more than doubles at 1-bit. The QADD mix has no images, yet MMMU-Pro moves with MMLU-Pro. At 3-bit the prefiller slightly hurts. Against a round-to-nearest NVFP4 prefiller, the trained IQ1S pair is still 13.1 points higher on MMLU-Pro, so the gain is not only the format swap. In llama.cpp, ODP cuts 8K time-to-first-token from 12.27 s to 6.90 s (1.78×). From 4K to 32K the speedup sits between 1.38× and 1.78×; shorter prompts stall on SSD load.
Format disaggregation via PTQ, no retraining, on models up to 2.8T: 11 of 13 model-benchmark pairs improve in point estimate, 6 at p<0.05, none degrade significantly. The deltas are about +0.1 to +1.1 points, a different scale from the 1-2-bit QADD gains.
Local 1-2-bit GGUF serving is where this is immediately usable: train a prefiller, leave the decoder alone, stream it from SSD, keep device weight memory flat, and get a faster first token on long prompts. Code, a llama.cpp fork, and the 27B prefillers are released.
In datacenter disaggregated serving the two checkpoints already live on separate accelerators, so full disaggregation is almost free and pays most at 2-3-bit decode. Format disaggregation is the cheap option when a second checkpoint will not fit: same weights, skip decode activation quantization, pick up 2-4 decode-heavy points.
This is incremental systems work, not a new architecture. Above 3-bit, on short prompts, and on MoE, the case gets thin.
The paper does not measure high batch, multi-turn, or agentic traffic. In multi-turn chat, cached assistant tokens keep decode-produced KV; rebuilding that history through prefill would write different representations. Cache-policy robustness is untested. ODP is a poor fit for MoE, because the load-to-compute ratio tracks the active-parameter fraction, and short-prompt TTFT is SSD-bound. Error bars are late-checkpoint jitter inside one run, not seed variance.
Several gaps sit outside that list. A 3-bit GGUF plus prefiller drops 0.6-2.8 points on the two 27B benchmarks; the headline lives at 1-bit, and there is no account of the negative transfer. The 27B distillation mix is math/code traces from the same BF16 model and overlaps a handful of MMLU-Pro prompts; training has no images while MMMU-Pro is a vision eval, so that transfer is unexplained. All speed numbers are DGX Spark at batch one. Whether prefill stays compute-bound, and whether ODP still pays, under high-concurrency vLLM is unmeasured. Prefillers also change generation length: IQ1S mean tokens on MMMU-Pro fall from 14579 to 6582 and truncation from 24.57% to 4.16%, while MMLU-Pro means get longer. Unchanged decode weights do not mean unchanged generation cost.