Factorized MLLM Embeddings for Composed Image Retrieval
WonJun Moon, Seongchan Kim, Dahun Kim, Seungryong Kim
cs.CV, cs.IR
2026-11-04
FactEmbed splits a composed query into 8 latents that imagine residual target details, then retrieves with one forward pass. Refined-CIRR R@1 is 69.0%, 1.58 points above ReCALL.
Composed image retrieval asks for a target image given a reference photo and a short edit, such as longer sleeves or a different logo. Users can point at the change more easily than they can describe the target from scratch.
MLLM retrievers usually read the last [EOS] state and align it to the target with the contrastive loss InfoNCE. A direct Qwen2.5-VL finetune and ReCALL both train that way. The loss only pulls two vectors together. It never requires the model to separate attributes the text changes from attributes it must keep. CCIN and FIRE-CIR move that split to an external language model at query time. The split is explicit, and so is the extra generation pass. FIRE-CIR's verification also grows linearly with the candidate list. Some zero-shot systems synthesize a proxy image instead.
FactEmbed keeps the split inside one forward pass: eight learnable latents simulate the edit, and the retrieval vector is formed on top of them.
The backbone is Qwen2.5-VL-7B-Instruct, updated with DoRA, a low-rank adaptation, at rank 16. Eight learnable latents sit in front of [EOS], read the reference image and the instruction through causal attention, and let [EOS] pool them into one global vector. The edit has to pass through those eight slots.
Contrastive learning alone drives the latents of one image toward a single direction. A light decoder, one self-attention block and a 3-layer MLP, is attached at layer 14 of 28 and reconstructs DINOv2 patch features of the target. Full-image reconstruction would spend the loss on low-frequency layout, so the target features are average-pooled to 2×2, upsampled, and added as a coarse layout prior. The latents only model the residual the prior does not explain.
Each latent predicts a spatial weight map and features. Softmax over the latent axis makes weights at one location sum to 1. The query side never sees the target image and must infer the residual from the reference and the text. The target side anchors its latents to the same factors so the two sets share slots.
A factor-level InfoNCE then matches the unordered sets with Sinkhorn optimal transport, entropy 0.1, 30 iterations. KL divergence also distills the decoder weights into attention at the third layer from the top, on the target pass only, since query attention sits on the reference grid. The shared backbone carries that local structure into inference. The global vector keeps an instance-level InfoNCE. Imagination is weighted 1.0 and distillation 0.5.
Imagination splits the latents, while the contrastive loss pulls them back into one point, so training is staged. Stage one optimizes imagination alone. Stage two turns on DoRA and the full loss. Stage three adds hard-negative triplets. The captioner, GPT-5.6-luna, also sees the original target and must write text that no longer describes it. That is 57,476 CIRR requests at $22.82 and 56,499 FashionIQ requests at $26.17. Near-duplicates with DINOv2 cosine above 0.99 are removed first: 743 pairs and 277 pairs.
The decoder is off at inference. One forward pass returns the global vector and the eight latents. The index stores the global vector and their mean. Retrieval uses the global vector, then adds the two cosines to rerank the top 100. The mean has no loss of its own.
The main table is Refined-CIRR and Refined-FashionIQ. Original labels are ambiguous, R@5 and R@50 saturate, and the paper treats R@1 as the stricter score. Refined-CIRR Avg averages R@1 and Rsubset@1 inside a six-image subset.
| Method | Refined-CIRR R@1 | Rsubset@1 | Avg | Refined-FashionIQ mean R@10 |
| Qwen2.5-VL finetune | 64.64 | 84.65 | 74.65 | 66.82 |
| CSMCIR (gallery captions) | 67.16 | 86.15 | 76.66 | 69.55 |
| ReCALL | 67.41 | 87.12 | 77.27 | 69.48 |
| FactEmbed | 68.99 | 87.45 | 78.22 | 71.13 |
Versus ReCALL, R@1 on Refined-CIRR is 1.58 points higher and mean R@10 on Refined-FashionIQ is 1.65 points higher, about 2.3% and 2.4% relative. CSMCIR adds a synthetic caption to every gallery image. FactEmbed does not, and its R@1 is still 1.83 points higher, about 2.7% relative.
On the original CIRR test set, R@1 is 56.60, against 53.76 for ReCALL, 51.98 for the plain finetune, and 53.88 for CSMCIR. R@5 is 84.87, and CSMCIR is slightly higher at 85.06. Subset R@1 is 81.86 versus 81.11. On original FashionIQ, mean R@10 / R@50 is 57.71 / 76.87, against 53.89 / 74.99 and 57.10 / 77.18. CSMCIR still leads R@50.
Qwen3-VL-8B reaches 59.25 R@1 on CIRR test, 2.65 above the 7B run. The ReCALL number cited for that same backbone change is 1.60. With Qwen3-VL-Embedding-8B, test R@1 is 60.43 and refined R@1 is 73.60.
Ablations on CIRR test R@1: plain finetune 51.98, imagination only 51.45, plus factor contrast 52.72, auxiliary losses and reranking without stage three 54.75, post-training on the baseline alone 53.69, full model 56.60. Against baseline-plus-post-training the gaps are 2.91 on original CIRR (56.60 vs 53.69) and 2.57 on the refined set (68.99 vs 66.42). Latents with no auxiliary loss, even when used to rerank, reach only 73.21 Avg on Refined-CIRR, below 74.65 without latents.
Full optimal-transport matching scores 56.84 and 69.20 R@1. Mean reranking scores 56.60 and 68.99, gaps of 0.24 and 0.21. On 2,297 CIRR validation images, one GPU, batch size 1, encoding takes about 353 ms and reranking adds 0.17 ms. Among decoder layers 7, 14, and 21, layer 14 is best. Contrastive training alone leaves a mean cosine of 0.841 among latents after the dataset mean is removed; imagination plus distillation lowers it to 0.548.
Editing a print or a sleeve is the product shape of CIR. An external language model on every query puts the latency on the request. The extra online work here is one added score over the top 100, and the index grows from one vector per image to two.
The gain is incremental. R@1 on the refined sets is about 1.5 to 2 points above strong CIR-specific models, and Refined-CIRR already sits near 91 at R@5 and 99 at R@50. The supervision is what travels: a coarse layout forces the latents onto residuals, and transport plus distillation returns that split to the global vector. The jump on Qwen3 is larger than the ReCALL backbone swap cited in the paper, so the loss is using compositional capacity the backbone already has.
The latent count is fixed at 8 and does not track scene complexity. Full-set matching is more accurate, and mean pooling drops some of the detail. The paper points to Matryoshka representations as a way to keep that accuracy with fewer stored vectors. Tests cover only CIRR and FashionIQ.
During training the query decoder sees a 2×2 pooled copy of the target features, so the residual is filled in after the coarse layout is known. Inference does not receive the prior, and no ablation removes it. GPT-5.6-luna both writes the stage-three text and judges whether a rewrite still fits the original target. The end-to-end comparison with ReCALL also uses a different data pipeline. With post-training on both sides, the architectural R@1 margin is 2.91 on original CIRR and 2.57 on the refined set. Alignment between latents and the edited attribute is shown only qualitatively.