Unpaired Modality-Agnostic Generative Recommendation
Weihao Shen, Wei Chen, Fuwei Zhang, Meng Yuan, Yuqin Lan, Guojun Liu, Qingsong Hua, Wei Lin, Fuzhen Zhuang
cs.IR
2026-08-04
UnpairGR shares one Transformer and residual codebooks across paired, image-only, and text-only items, lifting HR@1 up to 22.86% and staying robust under missing modalities.
Generative recommendation (GR) turns next-item prediction into text-like generation: every item gets encoded as a short sequence of discrete semantic IDs, and the recommender autoregressively generates the ID of the next item a user will want. How good those IDs are hinges entirely on whether they capture real item semantics. Recent multimodal GR methods build better IDs by mixing in images and text, but nearly all of them assume every item has both modalities cleanly paired. Real catalogs don't work that way: images come from sellers, descriptions come from a separate catalog pipeline, and coverage drifts out of sync. Some items end up with reliable photos but thin text, others with rich descriptions but no usable image. Existing tokenizers simply can't use the unpaired leftovers, not because the information is worthless, but because their quantization step only knows how to handle paired input.
UnpairGR's central bet is that the pairing bottleneck lives in quantization, not in the modalities themselves. Semantic IDs come from snapping a continuous representation into a discrete codebook cell; even a small shift in that representation can cross a cell boundary and produce a completely different, incompatible ID. If images and text pass through separate transformations, paired items and unimodal items drift into different corners of that space and end up with incompatible codes. UnpairGR keeps modality-specific processing to a thin input projection and shares everything downstream, one Transformer and one set of residual codebooks, across paired, image-only, and text-only observations. For paired items, the visual and textual representations aren't simply averaged; each is weighted by how confidently it lands in the shared codebook (measured via the entropy of its soft assignment), so the more decisive modality gets more say. Two regularizers keep everything compatible: one aligns paired visual and textual representations before quantization, the other keeps their post-quantization code assignments consistent, so the same item never ends up with contradictory IDs depending on which modality produced them.
On the fully-observed setting across Arts, Games, and Instruments, UnpairGR beats the previous strongest baseline, SynGR, by 22.86%, 17.14%, and 12.24% HR@1 respectively, with NDCG@5 gains up to 16.97 points. The more telling test is what happens under missing modalities: training and evaluating with only one modality present, UnpairGR still leads, by as much as 29.89% HR@10 on Games at 75% missingness. A separate diagnostic checks whether the method is actually learning one shared vocabulary rather than just improving accuracy by coincidence: comparing an item's paired ID against its image-only and text-only IDs shows 86.7-88.9% prefix agreement (the coarse codes match) but only 12-15% exact agreement (the fine codes diverge), which is exactly the coarse-to-fine pattern residual quantization is supposed to produce. On efficiency, UnpairGR trains 42% faster and serves 21% faster than SynGR on the Instruments dataset.
Real recommendation catalogs are never cleanly paired, and this gives teams running generative recommendation a way to use the leftover unimodal images and text they already have, without feature imputation, modality-specific codebooks, or fallback rules. It also solves cold-start for free: items with only one available modality get a compatible semantic ID from the same tokenizer, no separate handling required.
The ablation shows the shared Transformer is the single most important component, removing it drops macro-averaged HR@10 from 0.1477 to 0.1170, a bigger hit than removing shared codebooks or unpaired supervision. That suggests the method leans heavily on a backbone large enough to absorb cross-modality differences, and it's unclear whether the approach holds up with a smaller shared trunk. All three benchmarks are Amazon review categories with relatively clean images and text; the paper doesn't test how the reliability weighting behaves when the unimodal data itself is noisy, such as blurry seller photos or machine-translated descriptions.