Decode-aware calibration plus a bitmask SpMV kernel give pruned LLMs a real 1.48× decoding speedup

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang

cs.LG, cs.CL

2026-10-09

LLM pruning calibrated on the model's own decode-time activations (not fixed C4 text), plus a bitmask N:M SpMV kernel: up to 1.48× decoding speedup, near-doubled 2:4 scores.

What problem this solves

Autoregressive decoding is the memory-bound phase of LLM inference, and in long-output workloads it owns most of the latency. Training-free pruning (SparseGPT, Wanda and relatives) is the standard relief: use second-order information to zero out half the weights, cutting what gets read from memory at every decode step. Two problems undermine this line.

First, calibration mismatch. These methods collect activations on fixed text (C4, WikiText) under teacher forcing, where every step's context is handed to the model, and estimate the Hessian from those activations to decide which weights to cut. During real decoding the model consumes its own tokens; pruning errors at early steps rewrite later contexts and the activation distribution drifts. The paper measures the relative activation discrepancy between dense and pruned models: every layer hits 90% of its eventual plateau within the first 12 decode steps, then stays flat for hundreds more. The drift locks in almost immediately, and fixed-text calibration never sees that state.

Second, the kernels. cuSPARSELt and other 2:4 Sparse Tensor Core libraries target SpMM (the prefill shape) and deliver 1.31-1.46x there, but batch-1 decoding is SpMV territory, where the same 2:4 weights run at 0.85-0.87x dense throughput. Slower than not pruning at all.

Method

SparseDecoding moves on both axes.

Algorithm side:

• Draw prompts from the target task (LongWriter for writing, LiveCodeBench problem statements for code) and let the dense model generate continuations autoregressively
• Record input activations for every prunable linear layer at decode steps only; discard everything from prefill
• Assemble those activations into the calibration matrix and hand it to the unchanged SparseGPT solver

The solver is untouched; what changes is the distribution the Hessian summarizes, now the model's self-conditioned decoding states instead of teacher-forced text. Theorem A.1 formalizes the gap: the worst-case relative discrepancy between a calibration objective and the decode-time objective equals the spectral norm of the normalized Hessian difference. On Qwen3-14B/32B, autoregressive calibration has smaller discrepancy than C4 in at least 87% of prunable modules, with median ratios between 1.81 and 3.26.

System side, a Triton N:M SpMV kernel with two key designs:

• Bitmask indexing. The sparsity pattern becomes a bitmap, 32 columns packed per mask word; nonzeros live in column order so set bits map one-to-one onto stored weights and no per-nonzero column index is needed. At 50% sparsity this costs 2 metadata bits per nonzero, against 32 bits for nmSPARSE's int32 indices and 8 for a uint8 offset, a 16x and 4x reduction. Decoding is memory-bound; smaller metadata is direct savings.
• Fixed-step traversal. With M <= 32 at 50% density, every 32-bit mask word holds exactly 16 set bits, so each row's traversal length is a compile-time constant and the loop fully unrolls; a bit-scan instruction recovers the next nonzero's column each step. Every GPU thread does the same amount of work.

Cache policy is split too: the input vector uses .ca to benefit from L1 and L2 (it is reused across output rows), while weights and masks stream through L2 with .cg.

Results

WritingBench (1,000 prompts, six domains, DeepSeek-V4-Flash as judge, 1-10 scale) for writing; ClassEval (100 Python classes, class-level Pass@1) for code. Every comparison fixes the model, the SparseGPT backend, 50% retained weights and 1M calibration tokens; the only variable is where calibration activations come from.

Under 50% unstructured sparsity the four models gain 0.19 to 0.75 points on WritingBench. On ClassEval, SparseDecoding leads all eight model-sparsity pairs by 4 to 15 points; under 2:4, C4 calibration collapses to 0.0% on both Llama-3.1-8B and Qwen3-14B while SparseDecoding recovers nonzero scores. The advantage survives a Wanda backend, Pile and RedPajama calibration corpora, and the 8:16 pattern.

Throughput, measured end to end with GPT-Fast on A100 (batch one, context 512, median of 50 runs): Llama-3.1-8B goes from 96.3 to about 136.4 token/s (1.42x), Llama-3.3-70B from 21.7 to 32.1 (16:32 pattern, 1.48x), Qwen3-14B 1.35x, Qwen3-32B 1.45x. The four N:M patterns (2:4 through 16:32) land within noise of each other because all keep 50% of weights under the same bitmask format.

The gap to dense deserves honesty: Qwen3-32B scores 5.31 at 2:4 against 6.48 dense, and ClassEval falls from 34% dense to 24%. Cutting half the parameters still costs real quality.

Why it matters

The algorithmic change is nearly free for anyone already running SparseGPT-style pruning: no training, no solver changes, just collect calibration activations from the dense model's own decoding instead of fixed text. The payoff concentrates exactly where 2:4 hurts most, roughly doubling scores on both Qwen3 sizes. 2:4 is the format Tensor Cores natively support and the one engineering teams actually accept; part of its reputation for wrecking generation quality belonged to the calibration mismatch.

The kernel is useful independent of how the mask was chosen: any 50% N:M sparse weights can run decode through it. It also answers a familiar puzzle, why 2:4 pruning speeds up prefill yet slows down decoding.

Limitations

Calibration requires generating about 1M tokens with the dense model first, a real one-time cost. The gains partly come from matching calibration prompts to the evaluation domain (LongWriter for writing, LiveCodeBench for code); the paper never tests cross-task miscalibration. The cross-model ablation (pruning the 14B on the 32B's generations) differs by only about 0.1 points, so whose tokens matter little, but how much the task matters is unverified. All speed numbers are A100, batch one, context 512; longer contexts, larger batches and other GPUs are untested, and large-batch serving falls back to the SpMM path where cuSPARSELt already works. WritingBench scores come from a single LLM judge, so absolute values carry judge preference, though both arms share the judge. The paper has no dedicated limitations section; the points above are reader-side inference.

Terms

• Teacher forcing: feeding reference text token by token, so each step's context comes from the gold sequence rather than the model's own predictions.
• Hessian: the second-derivative matrix of reconstruction loss; pruning methods use it to rank weight importance.
• OBS (Optimal Brain Surgeon): classic second-order pruning framework that removes the least salient weight and compensates with the survivors; SparseGPT is its LLM-scale version.
• N:M sparsity: keeping N nonzeros in every group of M weights (2:4 = two of four); the semi-structured format NVIDIA Tensor Cores accelerate natively.
• SpMV / SpMM: sparse matrix-vector vs sparse matrix-matrix multiplication; batch-1 decoding is the former, prefill the latter.
• Bit-scan: a hardware instruction that finds the lowest set bit in a word, used here to recover nonzero columns from the bitmask.

Source

• arXiv 2610.12327v1
• PDF
• Hugging Face

Related papers

• Four LLM tutors intervene on 90% of problems at relative time 0.18, and transfer stays near zero
• SparseEngine hosts 15 sparse attention methods in one engine, up to 10x vLLM decode throughput
• EP-Flow predicts disordered crystals from the formula alone, at a 60.8% match rate
• 18 market encoders: similar rank IC, opposite latent geometry
• AI2D falls from 77.6% to 40.5% when multiple choice becomes free response
• Chess hack rate rises from 10.9% to 58.3% after number-only distillation

All paper explainers