Joint Constraint Aggregation for Qualifier-Sensitive Retrieval-Augmented QA
Jennifer Hsia, Zachary C. Lipton, Aarti Singh
cs.CL, cs.AI, cs.IR
2026-10-20
JCA checks candidates on shared constraints and compares the reports. It leads all nine hard slices; on Qwen3 the CoT gap is 7.1-20.6 points, then under 3 with reasoning.
Retrieval-augmented QA can still pick the wrong option when the right one is already in the evidence. A near-miss matches the role or the topic and misses a qualifier: a time, a version, a jurisdiction. Retrieval mentions that near-miss more often, so it looks like the best-supported choice.
Who plays Glinda in the original Broadway Wicked? The pages mention Ariana Grande in the film more often than Kristin Chenoweth in the original cast. A baseline picks Grande and drops Broadway. The score covers only the choice among candidates that already include the gold answer. Retrieval misses and generation misses sit outside this number.
On naturally retrieved AmbigQA pages, baseline accuracy falls as the gold answer is mentioned less often than the leading distractor. Holding the question, the candidate set, and the neutral pages fixed, and shifting support toward the near-miss, raises near-miss selection on Qwen3-30B in all 15 dataset-method conditions, by 1.4x to 9.7x at the endpoints. Support for a candidate on its own does not settle that choice.
JCA, Joint Constraint Aggregation, measures every candidate against one shared list of question constraints, then compares those measurements together.
A passage can support a person without showing that this person answers the question as stated. The NLI baseline scores the hypothesis that the answer is this candidate, using entailment probability minus contradiction probability. Candidates do not share constraints. Compare only tells the model to compare, and it writes no intermediate report.
Constraint Tally keeps the Step 2 verdicts and scores them +1, 0, and -1. Constrained NLI uses the same extracted constraints and still scores each candidate alone. Both land below full JCA in every reported setting. Collapsing the three steps into one completion lowers accuracy by about 4 to 10 points on the full sets and 10 to 18 points on the hard slices, for Qwen3-30B and Llama4-Scout. Checking every candidate in a single Step 2 call stays within 1 point of staged JCA on each backbone's pooled 2,018 full-set items.
All three settings use the top 20 Brave Search pages. AmbigQA distractors are human-annotated answers to nearby readings. AmbigQA-Extracted keeps candidates that Qwen3-30B pulls from the pages with no access to the gold answer, and only when the gold answer is recovered and another candidate group remains. On SituatedQA-temporal, the gold answer is the one for the stated time, and answers that are correct at other times are the distractors. The backbones are 4-bit Llama4-Scout-17B, Qwen3-30B, and Gemma3-27B. The hard slice is a mention gap below zero: some distractor is named in a larger share of the 20 pages than the gold answer. Full and hard sizes are 796/294, 548/198, and 674/238. AmbigQA and SituatedQA also require the gold answer and at least one distractor to appear in at least 10% of those pages.
On Llama4-Scout, JCA is highest on every full set. Direct, Compare, and NLI all trail CoT, the chain-of-thought baseline that reasons before it picks. JCA leads CoT by 4.0 to 11.3 points on the full sets and by 6.1 to 15.5 points on the hard slices. In Table 1, every paired comparison has a 95% bootstrap interval that excludes zero, except JCA against Constrained NLI on the SituatedQA hard slice. JCA is still ahead there, and the interval crosses zero.
| Setting | CoT full / hard | JCA full / hard |
| AmbigQA | 87.4 / 82.3 | 93.3 / 92.5 |
| AmbigQA-Extracted | 89.6 / 85.9 | 93.6 / 91.9 |
| SituatedQA | 74.2 / 64.3 | 85.5 / 79.8 |
JCA ranks first on the hard slice in all nine dataset-backbone cells. The widest gap is Qwen3-30B on temporal questions: 81.1 against a strongest primary baseline of 60.9. Gemma3-27B in the same cell scores 63.4 against 46.6. On the natural AmbigQA curve, baselines fall as the gap turns negative and JCA stays flatter. Twenty of 24 slope comparisons are not significant, so that curve does not rank robustness.
Step 3 helps when the verdicts tie or rank a distractor first. On the Llama4 full sets, joint aggregation over verdict labels alone beats the tally by 3.8 points on both AmbigQA settings and by 9.3 points on SituatedQA. Restoring the rationales adds 3 to 4 points on the AmbigQA settings. The extra 1.3 points on SituatedQA are not significant. When the verdicts already favor the gold answer, full JCA sits 0.5 to 3.2 points below the tally. On ties the Step 3 gain exceeds 30 points in all three settings, and AmbigQA reaches +35.5. When a distractor outranks the gold answer, the gain is 26.2 to 31.4 points.
Without reasoning mode, staged JCA on Qwen3-30B leads CoT by 7.1 to 20.6 points across the three hard slices. With Thinking-2507 on both sides, the gaps are +1.0, +1.0, and +2.9, and none is significant. On AmbigQA-Extracted, reasoning-mode JCA scores 91.4, below the instruction-mode 95.5, while CoT rises from 86.9 to 90.4.
With candidate recall held fixed, the choice still follows mention counts. Evidence for a near-miss can be true. It answers a neighboring reading. A longer chain of thought, or a prompt line that says to compare, does not cancel that count.
JCA is for the last pick, once several supported candidates are already in hand: time-scoped facts, entities that differ by version or role, the choice after a question has been disambiguated. If the gold answer never entered the set, the procedure has nothing to select. Merged checking is non-inferior to staged JCA inside a prespecified 2.5-point margin. On Qwen3-30B it cuts candidate-checking plus answer-stage input by 55% on AmbigQA and 49% on SituatedQA. Merge the checks to drop repeated context. Split them when the calls can run in parallel. The change is incremental. The gains sit mainly on instruction-tuned models, and mainly on items whose verdicts do not already name a winner.
The gold answer has to be in the candidate set. Items that extraction misses are dropped, and JCA leaves that recovery gap alone.
The hard slice counts alias mentions. It is a diagnostic cut, not a production failure rate. GPT-5.5 drops any item with a distractor judged co-correct, and some separable cases leave with the ambiguous ones. Recomputing the gap on only the text inside a context window changes membership.
The intervention shows that relative support changes the choice. Step 3 never sees the passages, and the aggregator can still use what the model already knows, so that gain was not separated from parametric memory. The corpus is one live-search snapshot. URLs, titles, snippets, and mention stats are released. Page text is not, and a later fetch can disagree. Larger candidate sets, and reasoning models other than this Qwen3 variant, were not tested.