Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking
Parinthapat Pengpun, Simran Khanuja, Graham Neubig
EMNLP 2026 Main Conference
cs.CL
2026-09-10
Wikidata rarity overlaps pageview tails by 37%. Pangea drops up to 39.9 points there. Qwen3-VL-8B searching Wikipedia hits 87.9% on MERLIN (+6.9) and +23.3 on rare slices.
Multimodal entity linking maps a marked mention, plus an image, onto a knowledge-base entry. Images often carry the signal that text withholds. A Japanese caption that only says "cruise ship" cannot be resolved until the hull in the photo reads Diamond Princess.
Current systems hold up on frequent entities and collapse on the tail. Prior work treated rarity as popularity: Wikipedia pageviews, incoming links. Popularity tracks how often people look something up. It does not track whether the entity is documented, connected in Wikidata, or present across language editions. An entity can be widely discussed and still thin in the graph, or obscure and well documented. The test set has both: 2016 Indian banknote demonetisation is a fat English article on a sparse graph; Muttahida Qaumi Movement is the reverse.
CMU NeuLab splits these notions on MERLIN, covering Hindi, Indonesian, Japanese, Tamil, and Vietnamese. Input is a passage, an image, and a marked mention. Output is the English Wikipedia title. Scoring is exact match.
Rarity is scored on two families of metrics from the Wikipedia and Wikidata APIs. Wikipedia metrics capture editorial attention: 90-day pageviews, backlinks, article size, revisions, unique editors, categories, external links, references, images. Wikidata metrics capture structure and cross-lingual coverage: in and out links, language editions, statements, qualifiers, entity age. An entity is rare on a metric if it sits in the bottom 5% of the test set. Those bottom-5% sets overlap only 37% on average; some pairs overlap 10%.
The linker itself is not trained. The backbone is Qwen3-VL in Thinking and Instruct variants at 2B, 4B, and 8B, so reasoning, retrieval, and scale can be isolated on one architecture. Retrieval runs over English Wikipedia, either BM25 lexical search or multilingual-e5-large-instruct embeddings in FAISS. At each step the model reads image and text, issues a query, reads snippets, and decides whether to search again, up to 20 rounds. Every RAG run forces the first search. In pilots, 2B models often answered without calling the tool, so a nominal RAG run collapsed to parametric guessing. After that first call, tool use is the model's choice. A second pass compresses the freeform trace into a single Wikipedia title.
Published baselines: GEMEL at 58.7%, mGENRE at 72.9%, Cultural Pangea-7B at 81.1% (then SOTA on MERLIN). A retrieval-aware extra, CulturalPangea-RAG, prepends the top-5 title-description pairs because Pangea cannot call tools.
Cultural Pangea scores 81.1% on the full set and drops 15.4 to 39.9 points on bottom-5% slices. The pageview slice drops 37.7 points; the Wikidata statement slice drops 37.0; the category-count slice drops 39.9. Structural tails and popularity tails are mostly different entities. A pageview-only audit misses a large failure set of similar size.
The best run, 8B-Think+Embed, reaches 87.9% overall, 6.9 points above Pangea. Hindi and Indonesian each gain 10.0; Japanese only 2.1. Redirect-aware scoring still leaves a 4.9-point lead. Fourteen of fifteen rare-entity gains beat the full-set gain.
| Slice (bottom 5%) | 8B-Think+Embed | vs Pangea |
| Full set | 87.9 | +6.9 |
| Qualifiers | 68.4 | +23.3 |
| Statements | 66.1 | +22.1 |
| Wikidata out-links | 66.3 | +21.7 |
| Language editions | 63.9 | +16.3 |
| Pageviews | 57.5 | +14.1 |
Reasoning and retrieval split cleanly. For 8B-Think, embedding retrieval is only +3.8 on the full set and +18.8 on the language-edition slice, about 5x. Think versus Instruct with no RAG is not significant on any rare split (p > 0.5). Instruct plus BM25 loses 4.9 points on the full set: it fires 3.2 to 3.7 searches per example with almost no thought between them, and by search 15+ verbatim repeats hit 34%, so noise overwrites parametric answers on common entities. On structural rare slices the same BM25 helps by 8.1 to 12.6 points, because there is no parametric answer to overwrite. Thinking models search 1.0 to 2.2 times per example, write 1,454 to 1,534 tokens between searches, and 56 to 58% of query transitions refine the previous query with extra disambiguation.
At 4B, Think+Embed hits 83.7%, matching 8B-Instr at 83.5% with half the parameters and about 2.8x the tokens. On rare entities the smaller reasoning model leads by 5 to 7 points. 2B-Think issues only the forced search on 97% of examples; tool use does not appear at that scale. Sticking retrieval onto Pangea drops it from 81.1% to 74.5%.
Of 841 residual errors, 72% never surface the gold entity in search. Another 23.5% see the gold entity in the trace and then reject it. Error rate is 31.1% on rare entities versus 8.4% on the head.
Three takeaways for people shipping multilingual entity linking or tool-using VLMs.
Popularity is the wrong rarity axis if the failure you care about is cultural long-tail coverage. Wikidata structure and language-edition count pick up entities that pageviews miss. MERLIN-Rare is released for that split.
Reasoning does not invent missing facts. Rare entities need retrieved evidence; reasoning's job is to search less, rewrite queries, and filter junk. Naive search on Instruct models hurts head accuracy. Wiring search into every model by default will damage common-entity scores.
A 4B reasoner with retrieval beats an 8B instruct model on the tail. The bill is tokens. The no-RAG 8B-Think baseline already holds 96% of the best system's full-set accuracy at 45% of the token cost. Full retrieval is not free on the head.
The factorial study is almost entirely Qwen3-VL. GLM-4.6V-Flash in thinking mode gains on all 15 rare slices when retrieval is added, but the full-set score falls from 84.5 to 81.8. Its non-thinking mode cannot keep a search loop alive: about 15% of examples collapse into token repetition. Cross-family evidence supports "retrieval helps rare entities," not the full reasoning-by-retrieval interaction.
The knowledge base is English Wikipedia. Every MERLIN entity has an English article by construction. Entities with no English page, and languages with thinner Wikipedia, including African languages, are untested. Cross-lingual retrieval remains the main bottleneck: a Japanese newspaper abbreviation 米 for the United States is read as a surname, the model searches "Mi surname," and the gold title never enters the candidate set.
Exact-match treats redirects and legal abbreviations as total misses. That matches prior MERLIN numbers and also marks some right-entity, wrong-string cases as errors.