Embedding Subspace Partitioning for Dynamic Multi-Objective Retrieval
Shaobo Zhang, Alice Leung, Yunxiang Ren, Ping Liu, Yuchin Juan, Qianqi Shen, Benjamin Le, Jianqiang Shen, Chengming Jiang, Ko-Cheng Wang, Vidya Krishnamurthy, Caleb Johnson, Fedor Borisyuk, Luke Simon, Jingwei Wu, Wenjing Zhang
cs.IR
2026-09-25
ESP partitions a bi-encoder into task-aware subspaces scored by serving-time weights. On LinkedIn job matching, retuning one model lifted revenue 4.0%, applications 3.95%, and cut no-fit 8.95%.
Industrial retrievers rarely serve one notion of relevance. LinkedIn job matching has to pack semantic fit, engagement, and revenue into the first-stage candidate set. A standard bi-encoder collapses those signals into one static vector and a single dot product. Bake the trade-off into the training loss and you retrain on every priority change; run one stack per objective and infrastructure grows linearly. The paper calls this the no-knob problem: operators have no in-product dial.
The constraint is geometric. As conflict between effective subspaces grows, the per-objective rank a shared embedding can preserve shrinks. Gradient surgery such as PCGrad eases optimizer fights. It does not remove the representation bottleneck.
ESP splits a d-dimensional embedding into k task-aware blocks and replaces the scalar dot product with a weighted sum of per-subspace similarities. Weights are applied on the request at serving time. When every weight is 1, scoring collapses to a normal concatenated dot product, so GPU exhaustive kNN still works.
The Transformer version reuses the model's native EOS token as a segment delimiter, puts the attribute segment first, then adds segment-aware attention masks and RoPE position resets. One forward pass isolates the subspaces with no extra parameters. The DNN version hangs a projection head per objective off a shared trunk. LinkedIn production uses the latter with k=3: engagement, revenue, and fitness.
Each subspace has its own loss. Engagement uses InfoNCE, revenue uses MSE on distilled pClick, fitness uses BCE. Output gradients do not mix across blocks. The shared trunk still sees mixed gradients, but the capacity guarantee sits on the output partition. Low-resolution objectives are cheap: entity matching saturates at 32 of 1024 dimensions, about 3%.
The open benchmark is MS MARCO with hard entity negatives: same entity, different information need. Kendall τ between semantic and entity rankings is 0.34, so the conflict proxy ε is 0.66. Theorem 3 then caps a 1024-d shared embedding at effective rank about 616, a 40% cut.
| Method | Semantic NDCG@10 | Entity Recall@10 |
| Multi-task single-head αe=1 | 0.924 | 0.887 |
| Multi-task single-head αe=10 | 0.823 | 1.000 |
| Single-head + PCGrad | 0.879 | 1.000 |
| ESP we=0.8 (serving-time) | 0.900 | 0.999 |
Pretrained ESP hits 0.843 semantic NDCG and 0.932 entity recall, near ColBERT, at one concatenated vector per document rather than O(L) token vectors. With a toy third objective (passage length) at weights (0.5, 0.3, 0.2) it reports 0.843 / 0.957 / 0.912.
Production is harsher. At matched revenue emphasis, single-head in-batch recall drops 28.24%; ESP drops 0.54%. The fitness head reaches AUC 0.938 with the other two metrics at k=2 parity. A 7-day A/B at the production point wd1=0.3 records +4.01% revenue, +3.95% applications, +0.92% click rate, and an 8.95% drop in no-fit. The same checkpoint spans four operating points with revenue lifts from +0.91% to +7.36%. The system serves 70M+ weekly users and has been in production for two years.
For retrieval teams, this moves multi-objective trade-offs from a training recipe to a serving knob. The price of admission is GPU exhaustive kNN, not IVFPQ-style ANN that assumes one coherent geometry. Multi-head retrieval wants one index per objective; ESP keeps a single concatenated index. Auxiliary business signals often fit in a compact block, leaving most dimensions for semantics.
This is an architectural change, not a new loss. Single-objective BEIR regression stays within 0.3 points. Without exhaustive retrieval hardware, the knob does not turn.
The weighted sum is incompatible with IVFPQ. The Transformer recipe depends on an upstream entity tagger and falls back to a single embedding when none is present. The open-source third objective is passage length, a toy; two high-resolution objectives colliding at once are still untested against the theory's worst case. Mixed gradients still flow through the shared trunk. The A/B reports significance without confidence intervals or traffic splits. On the MS MARCO hard negatives, entity and semantics are correlated, so semantic NDCG is almost flat for we in [0.3, 0.8] and only collapses to 0.583 at we=1.0. Complementarity papers over some of the conflict.