DoPR: Reusable Compressed Document Prefixes for Efficient LLM Reranking
Beiya Dai, Yifan Wei, Guang Yang, Xing Shi, Xinbing Wang, Zhouhan Lin
cs.IR
2026-09-03
DoPR precomputes query-independent document prefix KV states offline and processes only the query online. Document-side memory drops 8x, latency speeds up to 8.04x, and average NDCG@10 retention stays between 97.1% and 99.5% across Qwen3 scales.
Pointwise reranking scores each query-document pair independently, which parallelizes well and slots into existing retrieval pipelines. Its tax is redundant document-side computation: the same document gets retrieved by many queries over time, and a standard reranker re-encodes the full text for every pair. Prompt compression and token pruning cut the cost of one inference call but still reprocess the document per query. Precomputed document representations, as in late-interaction retrieval, live outside the LLM reranker's internal states. The open question is whether document computation can be reused inside pointwise LLM reranking itself.
DoPR moves document-side computation offline in three stages. A document-only forward pass averages final-layer self-attention over heads and scores each token by how strongly the document context attends to it, selecting the top-K salient hidden states with no trainable selector. During training, the input is arranged as [document; compressed representations; query; score token] under a structured mask that lets the query and score token reach the document only through the compressed representations, making them the sole information path, optimized end-to-end with a pairwise RankNet loss. At deployment, the selected representations are fed as prefix inputs and the resulting multi-layer KV states are stored; online inference injects them and processes only the query and score token.
The inference prefix budget decouples from the training budget, so quality-cost can be tuned without retraining. All experiments use a fixed 8x compression, retaining 12.5% of document tokens.
Across Qwen3 scales from 0.6B to 8B on TREC DL, BEIR, and BRIGHT:
| Setting | Quality retention | Doc memory | Latency |
| DL/BEIR 0.6B | 97.1% | 8.0x down | 1.21x up |
| DL/BEIR 8B | 99.5% | 8.0x down | 1.84x up |
| Covid 8B (long docs) | 100.1% | 8.0x down | 8.04x up |
| BRIGHT 0.6B | 98.6% | 8.0x down | 4.89x avg |
The pattern is clean: larger backbones tolerate the prefix bottleneck better, and longer documents buy bigger speedups, since with 256-token passages the bottleneck shifts to the query side and only 1.2x remains. The 0.6B model loses the most on Touche and Signal; at 4B and above results are near parity with occasional wins over the full-document baseline. On Llama-3.2-1B, retention is 97.5%, so the approach is not Qwen-specific. Attention-guided top-K beats first-K, uniform-K, and random-K selection at matched budgets.
For RAG and retrieval deployments, this pushes cross-query document reuse from the retrieval stage into the reranker's internals, turning KV states into an indexable asset. The 8x document-memory reduction is a concrete cost number for high-throughput serving, and the recipe holds from 0.6B to 8B. Tunable prefix budgets without retraining add deployment flexibility.
The benefit depends on documents being retrieved repeatedly; for fast-changing corpora the gain reduces to compression alone, which the authors state plainly. Offline construction and prefix storage are new indexing costs, and the store scales linearly with the catalog. The 0.6B model loses 3-9% out of domain, so small models are more sensitive to information loss. The 8x ratio is a global default; what compression discards, and whether the optimum varies by domain, is only probed indirectly through the Ktrain sweep. The reuse analysis uses a constructed workload, not production retrieval traffic.