Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
Zhengpei Hu, Kai Li, Dapeng Fu, Xuechao Zou, Yuanhao Tang, Yue Li, Tengfei Cao, Jianqiang Huang
cs.CL, cs.LG
2026-08-05
Hard prompt compressors keep the answer but drop the bridge linking it to the query. Six compressors break 30-60% of multi-hop examples; even GPT-5.5 loses 8.8 points.
Long contexts are expensive and poorly used. Self-attention is quadratic, prefill latency grows with length, and models underweight relevant content sitting in the middle of the prompt (the classic "lost in the middle" effect). Hard prompt compression is the common shortcut: before the text reaches the model, score every token, sentence, or chunk for importance and keep the highest-scoring units under a token budget. LLMLingua, Selective-Context, and Beaver all work this way.
The procedure hides an assumption nobody had formally named: that a retained fragment is usable on its own. It usually is not. A fragment can score high yet depend on an earlier definition, an antecedent, or a bridge fact connecting the query to the answer. Because scoring is independent, the compressor can keep the fragment and delete the very sentence it leans on. The authors call this referential dangling: the retained text is relevant but incomplete.
Figure 1 makes it concrete. The question is "What county was Tim DuBois born in?" The compressor keeps "Tim DuBois" and "Southwest City is in McDonald County," but drops the bridge "Tim DuBois was born in Southwest City." The answer string is still visible, but the inference chain from query subject to answer is severed. A relevance check passes; a dependency check fails.
The authors formalize the failure first. A compressed context dangles when it retains a task-relevant sentence but fails to retain, in full, any of that sentence's sufficient support sets, the minimal earlier text needed to interpret it and use it in an evidence chain.
Then comes a clean existence result (Proposition 1). Even under additive scoring, where each fragment gets an independent score and the budget picks the top-scoring set, the optimum can dangle while a complete alternative fits the same budget. The construction is three equal-cost fragments {x, d, z} with a budget for two; x is task-relevant and its only required support is d. Whenever u(x) > u(z) > u(d), additive selection picks {x, z}, which dangles, even though {x, d} is feasible and complete. The reason is structural: the additive objective has no term rewarding the joint retention of a fragment and the text it depends on. Dangling is intrinsic to independent scoring, not a bug in any one compressor.
For measurement, support annotations are incomplete, so the authors proxy the definition with directional content-word coverage: how much of a deleted definition's content words survive in the retained text, at a 0.5 threshold.
The practical piece is an automatic restoration pipeline. A candidate generator collects sentences from the deleted context that might support what was kept. A bert-base classifier scores each (retained sentence, candidate sentence, question) triple and reinserts the top K=3. Training pairs come from HotpotQA: positives are a retained sentence and a deleted annotated supporting sentence that share an entity; entity-sharing pairs that are not supporting are hard negatives, and non-sharing pairs are easy negatives. The hard negatives matter, because matching on entity co-occurrence alone would be fooled by incidental mentions.
Dangling is widespread. At compression ratio r=0.30, Beaver's dangling rate is 34.2% on HotpotQA, 53.5% on 2WikiMultiHopQA, and 54.2% on MuSiQue; the rate of retaining all annotated evidence is only 61.0%, 30.7%, and 27.0%. A manual audit puts precision at 95% (38 of 40 flagged examples truly needed the deleted paragraph).
Six different scoring signals do not escape it:
| Compressor | Scoring signal | Dangling rate |
| Beaver | embedding similarity (query-aware) | 32.1% |
| PartPrompt | syntactic parse | 47.8% |
| Selective-Context | self-information | 51.6% |
| LLMLingua-2 | learned token classifier | 56.0% |
| DAC | attention | 58.7% |
| LongLLMLingua | perplexity (query-aware) | 59.8% |
The two query-aware methods bookend the range, so query access is not the deciding factor. Deleted definitions are not low-scored: under Beaver similarity the mean salience percentile is 92.8% for definitions and 84.1% for answer sentences, both high. They get cut because under the budget something else scores a hair higher. On LongBench-v2 Single-Document QA, all 80 documents contain at least one dangling reference.
Does selecting different content at the same budget help? Substantially. Reinserting the deleted supporting paragraph and removing an equal number of tokens from paragraphs not annotated as support lifts Qwen3-8B accuracy by 28.8 to 34.3 points (p<10^-4) on the dangling subsets, recovering 88% to 92% of the gap to contexts that keep both supporting paragraphs. The reselected contexts are shorter on average, so the gain is not from more tokens.
Stronger answer models do not absorb the loss. On MuSiQue, GPT-5.5 scores 8.8 points lower on compressed contexts than on full-support contexts, and GLM-5.2 scores 24.2 points lower. On HotpotQA GPT-5.5 shows no aggregate gap, but decomposed it is four fixes and four breaks, not immunity.
The practical pipeline delivers a modest but real gain. With first-mention candidates, Qwen3-8B accuracy rises 4.7 points (p=0.022) while the compression ratio moves only from 0.30 to 0.31; Mistral-7B rises 6.5 points. If the annotated supporting sentence is included in the candidate set (an oracle upper bound), the gain reaches 8.0 points, which locates the bottleneck in finding the right sentence to reinsert rather than in the classifier.
Anyone running prompt compression inside RAG, agents, or long-document QA should know this: while you save money with relevance scoring, 30% to 60% of multi-hop samples are silently losing their evidence chains, and a larger answer model will not cover for you. The paper's contribution is not a new state-of-the-art compressor. It names and measures a hidden ailment that the field had tolerated without diagnosing: compression should optimize referential completeness, not relevance alone.
The direct engineering impact is moderate. The automatic restoration works on Beaver plus multi-hop QA, but it does not drop into token-level compressors (you cannot insert whole sentences into fragmented output), and it does not yet transfer to other compressors. Treat it as a diagnostic and a direction: in the short term, stay suspicious of compressed contexts on multi-hop and retrieval tasks, and compress critical chains less; in the long term, wait for dependency-aware selection to mature.
The authors concede two points: the restoration pipeline is calibrated only on Beaver and transfers poorly to other compressors, and candidate construction is the current bottleneck. A few more caveats remain. The measurement uses a content-word-overlap proxy at a 0.5 threshold rather than exact dangling detection, so it will diverge from ground truth at the margins. The whole diagnosis leans on multi-hop QA with support annotations; single-document QA needs a separate first-mention heuristic, and the two are not directly comparable. Proposition 1 is an existence result showing dangling is possible, not how frequent it is; the frequency claim rests on the six-compressor experiment. The targeted advantage of restoration is not fully nailed down either: random insertion of the same sentence count gains 2.0 points versus 4.7 for targeted, but the difference is not significant at this sample size (p=0.15). Finally, every result is confined to extractive hard compression; soft, gist-token, and KV-cache compression have different failure modes that this paper does not touch.