SA-RSQ: A Versatile Sparse Representation Framework for Multi-modal Recommender Systems
Xiang Wang, Shigang Quan, Tingzhen Chang, Kang Yang, Sitong Chen, Yabo Fan, Xingxing Wang, Zhaodian He
cs.AI
2026-08-24
SA-RSQ packs 2048-D MLLM vectors into 8-48 byte (index, prob) tuples. AUC hits 64.913 at 48 bytes; a one-week 10% A/B test lifts CTR 2.51% and CPM 3.66% vs no-multimodal.
Industrial recommenders want 2048-dimensional MLLM vectors, such as Qwen3-VL features, on catalogs of hundreds of millions of items. Storing the raw vectors blows memory: the paper’s 4096-byte dense features OOM the downstream DIN. The usual escape is RQ-VAE hard quantization into 8-byte Semantic IDs. Nearby items can land on different codes, distant items can share one, and training leans on a Straight-Through Estimator, so the tokenizer objective never matches CTR.
Soft quantization (SoftVQ-VAE) is differentiable, but it stores a dense low-dimensional vector. At 32 bytes in Float16 that is 16 dimensions, which is still a brutal 2048→16 bottleneck. The missing middle is a code that keeps distances better than hard SIDs and costs less than dense vectors.
SA-RSQ is residual soft quantization with Top-K routing instead of argmin. Residuals take a scaled dot product with the codebook, only the K largest logits survive, the rest go to −∞, then softmax. Reconstruction is a convex combination of those K codewords. Storage is K (Index, Prob) pairs: uint16 indices, Float16 probabilities. Shared codebooks are outside the per-item budget. Their accounting: index-only costs 2LK bytes, Index+Prob costs 4LK. L=4, K=1 is 8 bytes; L=2, K=4 is 32; L=4, K=3 is 48.
The support is discrete. Once it is fixed, probabilities and the weighted sum backprop without STE. Training starts dense (K=V) and cosine-anneals to the target K so gradients are not sealed on day one. The loss mixes global MSE, per-layer residual MSE, codebook orthogonality, and mutual-information regularization (sharp per-sample entropy, uniform batch marginals) to keep codes alive.
Two deployment paths. Two-stage: freeze sparse tuples offline, look up and weight online, keep the codebook trainable. End-to-end: sparse probabilities act as routing so CTR gradients can reach the encoder; the paper uses a light contrastive proxy for that. For generative recommendation they try Next-Distribution Prediction: predict the sparse codebook distribution with a KL loss, rather than autoregressive hard SIDs.
The corpus is a proprietary food-delivery ads set. Item semantics are 2048-D MLLM vectors. Every method is projected to a 16-D item vector for DIN. Metrics are AUC, gAUC, reconstruction MSE, and Semantic Cohesion. Splits and seed variance are not released.
At 8 bytes, index-only SA-RSQ (L=4, K=1) reaches AUC 64.616, above RQ-VAE 64.589, residual k-means 64.573, and R3-VAE 64.515, with reconstruction 0.2394 against hard-quantization numbers mostly above 0.33. At 32 bytes, Index+Prob (L=2, K=4) hits 64.836, above 16-D dense reconstructions in the same budget (about 64.4–64.7). At 48 bytes (L=4, K=3) the peak is AUC 64.913, gAUC 63.013, reconstruction 0.1553.
Ablations at the 48-byte setting: drop MIR and codebook usage falls from 100% to 24.5%; drop curriculum and usage is 15.9% with reconstruction 0.4942; drop the Top-K mask (dense soft quantization) and AUC is 64.604. Full-model Semantic Cohesion is 0.9179; removing proxy alignment drops it to 0.1014 and AUC from 64.913 to 64.746.
The generative study is a probe. Under NTP hard SIDs, SA-RSQ’s discrete indices reach R@10=0.0088 versus RQ-VAE 0.0068 and R3-VAE 0.0075. NDP moves R@10 to 0.0096. Absolute recall is tiny; the authors do not sell this as a new paradigm.
The online test is one week at 10% traffic against a production baseline with no multimodal features, reporting each method’s peak config. SA-RSQ L=4, K=3: +2.51% CTR, +3.66% CPM. The same table gives RQ-VAE 4-layer SID +0.96% / +1.27% and R3-VAE 24-D dense +0.85% / +0.97%. Traffic counts, intervals, p-values, and latency are withheld.
| Budget | Method | AUC | Recon MSE |
| 8B index | RQ-VAE SID | 64.589 | 0.4010 |
| 8B index | SA-RSQ L=4,K=1 | 64.616 | 0.2394 |
| 32B tuples | SA-RSQ L=2,K=4 | 64.836 | 0.1660 |
| 48B tuples | SA-RSQ L=4,K=3 | 64.913 | 0.1553 |
This is a tunable middle representation for MLLM features in recommenders: bytes decouple from codebook dimension, and distances stay in a continuous space. It ran for a week in food-delivery ads. Offline AUC moves in the fourth digit; the 2.5% relative CTR lift is the commercially meaningful number, against “no multimodal,” not against the strongest equal-byte system. Teams already on SIDs, stuck on boundary distortion, and unwilling to serve raw 2048-D vectors are the audience.
The data are private. User/item counts and splits are withheld, so public-benchmark checks are impossible. Main tables are single point estimates. The A/B test has no significance numbers and no p99 latency, so peak configs are deployment anecdotes. The prose once writes the 8-byte setting as L=1, K=4 while the table says L=4, K=1; both cost 8 bytes, and the implementation detail does not line up. NDP recall stays around 0.01, enough to show a probabilistic target can train, not enough to beat mature SID generation. How orthogonal loss and MIR transfer to other modalities is open.