SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
Zirong Chen, Fuda Ye, Kuan Zhang, Enjun Du, Junfu Pu, Xinlei Wang, Xinyu Zuo, Lisheng Duan, Jin Ma, Yongqi Zhang
EMNLP 2026
cs.CV, cs.IR
2026-08-30
SnapBench tests 16 retrievers on 1,145 queries and 53 corruptions. Image noise hurts most; joint retrieval trails image-only by 11.21 R@1. MOOR barely beats whitened image-only.
A phone camera points at an object and the user asks a short question such as "what flower is this". The retriever has to rank the intended entity above same-category lookalikes. Photos are blurry, cropped, watermarked; questions are short, coarse, or mistyped. Existing retrieval sets use clean captions, or they perturb one modality and change the item under test. That cannot isolate "same intent, same gallery, only the snap got worse".
SnapBench is a paired test for that setting. Clean and corrupted queries share the target, the gallery, and the labels. The work comes from HKUST (GZ), Tencent Yuanbao, Tsinghua, and HKU, and is aimed at EMNLP 2026.
Query images are screened from a web pool so each has a clear primary entity. Gemini-3-Flash writes a coarse English question; rule checks plus GPT-5.4-mini filtering leave 1,145 queries. Gallery items are sampled from the same pool with no overlap, captioned by a private Qwen-VL model, then pre-retrieved to a top-30 for annotation. Ten annotators score visual similarity, entity relevance, caption accuracy, and intent fit on a 0-3 scale. Fitness at least 2 is a positive, 1 is a hard negative, 0 leaves the gallery. The final gallery has 9,085 items, 4.77 positives and 3.50 hard negatives per query. Annotators needed 90% calibration accuracy; the final audit pass rate is 95%.
Corruptions hit the query only. Image side: 15 operators in Add, Remove, Degrade, Transform, three severities, 45 conditions. Text side: 8 rule-based operators (character edits, word repeat or swap, a chit-chat prefix, replace with "What is this?"). Six joint image-text pairs are tested as well. Total: 53 controlled conditions.
MOOR is a training-free fusion probe. It whitens four similarity paths (image-image, text-text, image-text, text-image), gates the text-involving paths by Pearson correlation with the image-image scores through a bell function, and weights by score variance. No learned parameters.
Clean R@1 across 16 models runs from 39.8 (SigLIP2-SO400M) to 79.1 (Ops-MM-7B). Nobody clears 80. VLM embedding models beat dual encoders by 13 points on average.
Image corruption drops mean R@1 from 63.2 to 57.9. Severity s1 / s2 / s3 cost 0.8, 3.8, and 11.2 points. At s3, Remove is worst (16.3), then Degrade 12.5, Add 10.9, geometric Transform 5.2. Clean rank does not predict robustness: SigLIP-SO400M looks strong when clean and falls hard under severe corruption.
Text corruption barely moves joint retrieval (63.5 / 63.7 / 63.9 for char / word / sent). That is underuse, not robustness. Joint IT→IT trails image-only I→I for most models, mean drag 11.21 R@1. On GME-7B, swapping "What flower is this?" for "A tulip." cuts the drag from 7.68 to 0.17, a 97.8% wipe. Coarse wording is the cause.
Joint image-text corruption does not add. Across six pairs the observed drop is 13.6 against a naive 6.0, a +7.6 super-additive gap. GME shows the largest overshoot; Ops-MM and VLM2Vec are sub-additive.
MOOR averages 69.80 R@1 over 54 states and 16 models, above whitened uniform fusion at 60.72 and RRF at 45.64, and only 0.43 above whitened image-only at 69.38. The oracle sits at 80.07. CLIP gains about 6.0 on average; Qwen3-VL-2B gains 0.5. The paper says MOOR closes 40% to 60% of the oracle gap, with dual towers gaining more.
Snap-and-ask is already a default mobile entry, and most retrievers still use a fixed fusion weight. A coarse question lifts every same-category hard negative and smears a clean visual ranking. For product work, "image plus text is always better" fails here. Image-only is the safer default; text should gain weight only when it actually discriminates.
MOOR is useful as a diagnostic for overweighting a weak modality. It barely beats whitened image-only, which is itself the finding: on this benchmark, good calibration is close to trusting the photo.
Corruptions are synthetic, not crawled blurry uploads. A 5,000-upload production sample found at least one issue on 71% of pairs and both modalities on 35%. A 200-image real-corruption check still ranks Degrade and Remove as the most damaging (R@1 drops of 9.8 and 12.4). The joint distribution in the wild can still differ.
The set is small by design: 1,145 queries in exchange for dense labels and hard negatives, not web scale. Questions are model-written then filtered, so they will not match real typing. MOOR reverts to fixed fusion when both modalities are uniformly good or uniformly bad. The eval stops at retrieval and does not score the later answer.