MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval
Seokwon Song, Sohyeon Kim, Gunhee Kim
EMNLP 2026
cs.IR
2026-08-31
A 104.9K Stack Exchange benchmark shows multimodal retrievers collapse to one viewpoint. SPIN injects 20.5K noise parameters and lifts Qwen3-VL HC@10 from 25.6% to 37.8%.
Open-ended questions hide several viewpoints. "Why are so few foods blue?" can be pigment chemistry or cultural psychology. Most IR benchmarks are closed factoid items. Even open-ended sets usually ground each query in one subject and one modality.
Classic diversification is expensive. Query expansion calls an LLM per request. Multi-vector training wants document-level labels. This EMNLP 2026 paper from Seoul National University ships both a benchmark and a nearly freeze-the-backbone fix.
Multi3IR takes posts from 77 Stack Exchange sites with at least four answers. GPT-5-mini extracts perspectives; near-duplicates above cosine 0.8 are dropped, and MiniCheck filters non-entailed items. Each perspective queries Google Images and C4 for top-10 hits; Qwen3-VL-30B keeps only documents that support that perspective alone. Subject domains come from EAI-Distill-0.5b.
Scale: 104,916 queries, 521,739 perspectives, 1.01M documents. Averages per query: 4.98 perspectives, 3.34 domains, 1.91 modalities. Human checks on a 1,000-query test split: 99.1% relevant, 96.2% unique, 90.2% exclusive support.
SPIN freezes the retriever. At a middle layer it adds m learned noise vectors to the query hidden state, then runs the rest of the stack, producing m query embeddings. Targets are embeddings of perspective descriptions, not documents. A noisy-OR positive loss asks that every perspective be covered by at least one vector; a negative loss asks every vector to reject in-batch perspectives from other queries. Inference retrieves m lists and merges them by round robin. Injection is at 0.5L (layers 16, 14, 18 on the three backbones). m=5. The document tower stays frozen.
Zero-shot retrievers pile onto one dominant perspective. Feeding gold perspective texts and merging by round robin recovers the missed documents, so the bottleneck is the query vector, not the document space.
Qwen3-VL-Embedding-8B on the 1.01M pool:
| Method | HC@10 | SC@10 | Supervision |
| Naive | 25.55 | 36.22 | none |
| Naive + doc FT | 28.06 | 41.41 | documents |
| LLM expansion | 25.33 | 37.75 | perspectives |
| ARE | 33.36 | 44.73 | documents |
| SPIN | 37.83 | 59.35 | perspectives |
| Oracle perspective queries | 52.31 | 70.04 | — |
On MM-Embed, SPIN HC@10 is 35.24 versus 28.83 naive; on GME-Qwen2, 34.02 versus 21.81. Under the same perspective labels, SPIN beats LLM expansion by 8 to 13 HC@10 points. Document-supervised ARE trails SPIN by about 4 HC@10 and 14 SC@10.
The same steering transfers to PIR and BeRDS. PIR HC@10 rises about 9 to 12 points; on BeRDS, GME moves from 64.95 to 85.05. Mid-layer injection is the lever: L18 keeps climbing with m, L30 plateaus. SPIN's 20.5K parameters beat LoRA (105M) by 7.5 HC@100. Modality coverage at k=100 is 78.2 versus oracle 81.1; domain coverage is 68.3 versus 84.8. On the most dispersed positive quartile, m=5 adds 24.4 HC@100 over naive.
The failure mode is now named: documents are reachable, the query vector only represents one reading. SPIN trains against perspective text and turns a frozen encoder into a multi-vector retriever without document labels or an LLM rewrite at serve time.
Twenty thousand parameters on a frozen backbone is a patch you can put on a production retriever. The oracle remains well above, so perspective discovery itself is still open. Domain coverage lags modality coverage; the next gap is "search another field", not "rephrase the same field".
m is fixed at 5 while gold perspective counts run from 4 to 9, with no per-query adaptive budget. SPIN never uses the supporting documents in training; joining descriptions with documents may still help. The pipeline is automatic, so LLM bias can enter the labels; soft coverage is judged by GPT-5-mini. The test split is human-checked, the train split is not. Retrieval is measured inside a 1.01M gold pool, not the open web.