Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Hangrui Xu, Zhengxian Wu, Yunyao Yu, Zhuohong Chen, Rui Cong, Xiangwen Deng, Zhifang Liu, Peng Jiao, Haoqian Wang
cs.CV
2026-08-20
KBMR is an MLLM retriever for KB-VQA: InfoSeek Recall@1 rises 14.7 points over EVA-CLIP-8B, and end-to-end VQA gains reach +9.4 when only the first-stage retriever is swapped.
Knowledge-based VQA asks questions the pixels do not contain: who owns this hotel now, how this bird forages in gardens. The usual fix is RAG. Retrieve the same entity from a Wikipedia-scale image-text store, then let a multimodal LLM answer from that evidence.
First-stage retrieval is almost always a CLIP dual encoder. CLIP is good at "looks like"; KB-VQA needs "is the same entity". One building changes with time of day; two crow species look alike. If the candidate pool is wrong, later rerankers, reflection tokens, and query tools are searching a bad set. Wiki-LLaVA, ReflectiVA, and Wiki-PRF mainly add modules after retrieval. The retriever itself is still CLIP.
KBMR swaps the retriever for an MLLM embedding model, claimed as the first such retriever for KB-VQA. Training has three parts.
Representation: the MLLM is prompted with "Summary above image in one word:" and the last-token hidden state is the retrieval vector. Vision and the instruction share one autoregressive semantic space, unlike a CLIP dual tower.
Supervision: Wikipedia-scale positives and negatives are noisy. EVA-CLIP first pulls the top-50 visual neighbors of each query (true positives removed) as a potential hard-negative pool. A frozen MLLM Semantic Discriminator (SD) then answers whether Candidate and Query are the same entity. Yes/No logits become a continuous entity-consistency weight w=σ((zyes-zno)/γ). Candidates above the query-positive weight minus a margin are dropped. The rest are split into four difficulty strata by weight, two samples each, eight hard negatives per query.
Training: standard contrastive learning treats the positive as 1 and everything else as 0. Hard negatives in KB-VQA are not equally irrelevant. KBMR turns both SD weights and retrieval similarities into temperature softmaxes, then pulls the retriever posterior toward the SD prior with symmetric KL. That objective is called continuous semantic distillation. γ is 1.1.
The SD is Qwen2.5-VL-7B. Retrievers are Qwen2-VL-7B and LLaVA-OneVision-7B, LoRA rank 16, 8xA100, 336x336, accumulated batch 1024, 600k samples, 5,000 steps. At inference only the trained retriever runs similarity search.
Retrieval: on E-VQA, KBMR (LLaVA-OV-7B) reaches R@1 24.7, 11.4 points above the strongest CLIP (EVA-CLIP-8B at 13.3) and 12.8 above the strongest zero-shot MLLM retriever. On InfoSeek, R@1 is 60.3, 14.7 above EVA-CLIP-8B at 45.6 and 19.0 above the best zero-shot MLLM. Zero-shot MLLM embeddings generally trail strong CLIP, so "can see" does not imply "can retrieve entities".
End-to-end, only the first stage changes. EchoSight with reranking moves from 41.8 to 51.0 on full E-VQA (+9.2). OMGM reaches 54.7 / 50.8 on E-VQA / InfoSeek, new highs on both. MMKB-RAG's Unseen-E on InfoSeek goes 36.3 to 45.7 (+9.4), the largest end-to-end gain quoted in the abstract. Plugging KBMR into OMGM on OK-VQA lifts the VQA score from 66.6 to 79.3 (+12.7), above Wiki-PRF-7B at 77.8.
Ablations, all with eight hard negatives: random + one-hot scores 9.1 R@1 on E-VQA; CLIP hard negatives + one-hot 14.7; SD hard negatives + one-hot 16.4; SD hard negatives + SD soft weights 24.7. Flipping 30% of SD weights drops the score to 12.1, worse than one-hot. SD AUC for same-entity pairs is 0.91 versus 0.79 for CLIP similarity. Eight negatives beat 4 and 6; 10 does not help. Qwen2.5-VL-7B as SD slightly beats InternVL3-14B.
| Retriever | E-VQA R@1 | InfoSeek R@1 |
| EVA-CLIP-8B | 13.3 | 45.6 |
| LLaVA-OneVision zero-shot | 7.4 | 21.8 |
| KBMR (LLaVA-OV-7B) | 24.7 | 60.3 |
In a KB-VQA stack, the first hop is worth more than another reranker. The paper moves entity alignment into the retrieval space and replaces 0/1 contrast with graded weights, which is the right failure mode for long-tail entities with large appearance change. The engineering claim is plug-in: EchoSight, ReflectiVA, and OMGM all rise after a retriever swap.
The cost is also clear. The retriever is a 7B MLLM, roughly an order of magnitude heavier than CLIP at inference; latency and throughput are not reported. Training hangs a second 7B SD as teacher. This buys a better candidate pool with compute. It is not a free lunch.
There is no standalone limitations section. Several issues sit in the setup.
The hard-negative pool is still mined from EVA-CLIP's top-50, so entities CLIP never ranks close never reach the SD. The knowledge base used for evaluation is 100k Wikipedia entries, not the full dump. Images are resized to 336x336, which can erase fine species cues. 600k training samples are drawn from E-VQA, InfoSeek, and OK-VQA train splits; InfoSeek's val set is disjoint in entities and questions, E-VQA reports the test set, but entity-level leakage is not audited. MMKB-RAG numbers are a reimplementation. The SD itself is wrong some of the time (AUC 0.91), so its bias is distilled into the student. The "first MLLM retriever for KB-VQA" claim depends on the related-work cut. General MLLM embeddings (VLM2Vec, GME, MM-Ret) already exist; the difference here is the task head and the distillation target, not the idea that an MLLM can emit a vector.