Is Personalized Modality Weighting Actually Personalized? A Controlled Audit of Per-User Weighting Claims in Multimodal Recommenders
Jingyuan Zheng, Xin Zhang, Yang Gu, Dongjing Wang, Yuxiang Wang, Xudong Shen, Haiping Zhang, Youhuizi Li, Dongjin Yu
cs.IR
2026-08-06
An audit of six per-user modality-weighting heads on three short-video corpora finds a single global weight already captures nearly all content gain, per-user weighting adds no consistent utility, and the apparent personalization signal is architectural confounding.
Multimodal recommenders popularly give each user a set of modality weights: some trust visuals more, others text or audio. The idea appears as user modality-strength vectors, attention gates, meta-weight hypernetworks, and low-rank guided weights, each claiming a ranking gain over global or no weighting.
The authors flag an overlooked issue: prior evaluations never separate a genuinely user-specific signal from a single global weight plus extra model capacity. Standard ablations remove a modality or module but do not cut the binding between a user and the weight they receive, so a gain a single global weight would already produce gets counted as personalization. Two confounds stay uncontrolled: a global modality scale that helps every user equally, and the extra capacity a per-user head adds to the backbone.
This is an audit paper built on a two-contrast principle run on one shared collaborative-filtering backbone.
The first contrast is the utility gap (real-GM): a per-user head against a single global weight (the strongest non-personalized alternative), where only a positive utility gap counts as useful personalization. The second is the identifiability gap (real-shuf): at evaluation time each user's weight is swapped for another user's from the same activity decile (the trained head is frozen, permutation happens only at eval), testing whether the binding matters.
Six heads are audited: a free per-user table (PUM), a bilinear attention gate (ATT), a meta-weight hypernetwork (MWN), low-rank guided weights (LRG), plus decoupled variants ATTd and MWNd where the gate reads a private embedding. Three independent short-video corpora: Tsinghua ShortVideo, KuaiRand-27K, MicroLens-100K, with Amazon-Baby as a cross-domain replication. A signal-implant calibration plants modality preferences at varying strengths and requires capture AUROC to rise monotonically, proving the harness can detect user-specific structure. Five random seeds, paired t-tests, Benjamini-Hochberg FDR correction.
A single global weight already captures nearly all of the content gain.
| Corpus | Global weight over no-modality baseline (PairAcc) |
| Tsinghua ShortVideo | +1.89pp |
| KuaiRand-27K | +3.61pp |
| MicroLens-100K | +3.50pp |
All at p<.001. Made per-user, none of the six heads beats the global weight across all three corpora and three metrics, and the few positive gaps are small (at most 0.9 points) and flip across corpora.
The sharpest finding is a dissociation: on KuaiRand-27K the attention head posts an identifiability gap of 128% of the content gain (swapping weights costs 4.64 points), yet its utility against the global weight is negative (it trails by 0.7 points). A head can look strongly personalized and still lose to a global weight. The authors trace the cause: the gate reads the shared collaborative embedding. Replacing the gate input with an independent embedding collapses the inflation from 128% to 0.6% (attention) and from 87% to 0.2% (hypernetwork), while the utility conclusion stands. Signal-implant calibration gives capture AUROC of 0.57 to 0.64, so the null is a genuine absence of user-specific signal, not a measurement failure.
It throws cold water on a popular family of claims: per-user modality weighting shows no stable personalization benefit in evaluation, and much of the past "significant shuffle control" rests on architectural confounding. The prescription is concrete: a claim of effective per-user modality weighting should report real-GM greater than zero with statistical significance on at least two independent corpora, and calibrate the harness with a signal implant before trusting any null.
For recommender practitioners, this is a reusable two-contrast audit protocol and a reminder: do not only compare a head against its own shuffle.
The authors bound the scope in three ways themselves. It covers static user-weight binding in the weighting family, not fusion architectures or session-level dynamic weights; the single-backbone reduction assumes the family differs in the head rather than the collaborative path (supported by a LightGCN backbone check). The audit is on reimplementations under one backbone, not original author code, so claims concern the weight-production mechanism under matched conditions, not any original system's reported numbers. The main results rest on three short-video corpora with implicit-feedback proxy metrics, and pairwise accuracy stands in for a modality-driven preference that cannot be observed directly.
Readers should also note: what is falsified is personalization for this implementation family at this data scale, not the existence of user modality preference itself. Capture AUROC of only 0.57 to 0.64 means weaker signals could be missed.