LegalPincite: Multi-level Legal Information Retrieval Dataset
Theresia Veronika Rampisela, Henrik Palmer Olsen, Giovanni Colavizza
cs.IR
2026-08-04
Legal IR datasets leak the answer in the query and drop non-cited paragraphs, inflating scores. LegalPincite masks citations, keeps all paragraphs, tests three retrieval levels.
The core legal-IR task is finding precedent: given a paragraph being written in a judgment, find which past cases or paragraphs to cite. In practice lawyers usually want not which case but which paragraph within it; that pinpoint citation is a pincite. Public legal-IR datasets fail this in two ways. Most support only case-level or paragraph-to-case retrieval, so pincites are impossible. The few that handle paragraph level (all built on EU Court of Justice, CJEU, judgments) are dirty in two specific ways: the query paragraph leaks the answer, carrying the cited case number and paragraph number verbatim, and the corpus keeps only citing or cited paragraphs, discarding the majority that neither cite nor are cited. That shrinks the candidate pool artificially and inflates retrieval scores.
LegalPincite merges and repairs two existing CJEU datasets, then adds 2021 to 2025 judgments. The work is engineering cleanup plus masking: detect paragraph language and drop the roughly 5% that are not English; recover 268 paragraphs of missing text; re-segment with EUR-Lex's Cellar API to fix parsing errors across 1,400+ cases. A legal NER model plus regular expressions strips the answer-leaking cues from queries (case title, case number, paragraph number, parties). The corpus keeps every paragraph of every case, not only the citing or cited ones, so the candidate pool mirrors reality. A chronological split (train pre-2018, dev 2018-2021, test 2022-2025) forbids a query from citing a later precedent, preventing temporal leakage. Two legal experts annotated relevance on a subset. Everything ships as pyterrier-compatible CSVs across three retrieval levels.
Only four bag-of-words baselines were run (TF-IDF, BM25, LMIR, DPH); the authors note they lacked the compute for dense retrievers on long legal text. On the test set, by retrieval level:
| Level | Best baseline | NDCG@10 |
| case-to-case | LMIR | 0.442 |
| paragraph-to-case | TF-IDF | 0.555 |
| paragraph-to-paragraph | BM25 ( TF-IDF) | 0.528 |
The leakage ablation is the telling part. With all citation info stripped from queries, paragraph-to-paragraph NDCG@10 is highest, beating the leaky original by 34.7%, which means the leaked citation text was acting as noise. But removing only the paragraph ID is not enough: case-level scores still inflate by 35.6% to 55%, so the authors recommend the fully masked queries. BM25 is unstable on long legal documents because it over-penalizes length, consistent with earlier findings.
For anyone building RAG, legal NLP, or long-document retrieval, this is a clean, leakage-free benchmark with paragraph-level labels that reflects real difficulty. The practical hook is the pincite: judgments run to hundreds of paragraphs and a lawyer usually needs one specific paragraph, so a paragraph-level system is far more useful than a case-level one. It also quantifies a quietly important effect: many existing retrieval highs are propped up by leaky queries, and the true difficulty only shows once the answer is masked out.
The authors list several. Ground-truth citations come mostly from EUR-Lex, so there is a feedback loop: judges may have found these very citations through a retrieval system, meaning "cited" is not the same as "most relevant." Expert annotation covers only the top-10 of a single dense retriever, so it is not exhaustive; evaluation should favor precision at short cutoffs over recall. The automatic masking pipeline has manually confirmed misses, with a systematic audit left to future work. One more caveat: with only four bag-of-words baselines and no dense retriever, the ceiling on this benchmark has not been probed; the numbers show only the relative strength of lexical methods, not where modern retrievers actually land.