The Case Against Generation for Retrieval: Discriminative Language Models as Effective Retrievers
Zhe Xu, Prachi Agrawal, Kavosh Asadi, Tianyi Chen, Carl Hu, Justin Johnson, Wuwei Lan, Mingfu Liang, Xi Liu, Tik On Lui, Oladipo Ositelu, Sandeep Pandey, Ankit Peshin, Feng Qi, Anil Ramakrishna, Kaushik Rangadurai, Frank Shyu, Luke Simon, Yang Yang, Chiyu Zhang
cs.IR
2026-07-28
Meta argues against generative retrieval, using the LLM as the two-tower encoder's backbone instead. It matches production with 0.5% of training data and lifts public R@10 by 21%-64%.
The instinctive way to put an LLM in charge of retrieval is generative: have it autoregressively emit an item identifier. That path has two hard problems. Serving cost: token-by-token decoding is slow and expensive at web scale. And grounding: the tokens the model generates may not correspond to any real item in the catalog, so you need extra constraints or a re-rank step.
Meta's position is in the title: do not let the LLM generate, let it be the semantic encoder inside a classic two-tower retrieval architecture. Two-tower is the workhorse of retrieval (a user tower and an item tower each emit a vector, ranked by inner product); it is fast and its item vectors can be indexed offline, but its expressiveness is capped by the encoder. Dropping an LLM in as that encoder buys the LLM's semantic understanding while keeping two-tower's serving efficiency.
The design is a cross-encoder teacher distilling into a two-tower student, both built on a shared LLM encoder.
Teacher-side improvements:
Student-side (two-tower) components:
On public Amazon datasets (Beauty, Sports, Toys), the cross-encoder teacher reaches SOTA, with R@10 up over the ORT baseline:
| Dataset | CE teacher R@10 | vs ORT |
| Beauty | 0.0957 | +21.0% |
| Sports | 0.0677 | +64.3% |
| Toys | 0.1223 | +53.5% |
The two-tower student keeps up: Sports R@10 0.0542 (+31.6%), Toys 0.1078 (+35.3%).
The most consequential ablation is CE2TT distillation; removing it drops R@10 by 13.3%, 23.1%, and 8.0% on the three sets.
The internal Meta data is the most persuasive part: the LLM-native two-tower matches the production DLRM baseline's normalized entropy (NE) using only 0.5% of the training data, and the cross-encoder version cuts NE by a further 2.25%. Frozen, the model still holds +2.21% to +2.25% NE while the comparison DLRM degrades by -2.68% to -4.30%, so the LLM encoder generalizes clearly better. Tail-item NE improves 5.5% and head-user NE 2.3%.
This points the LLM-in-recommendations effort at a path that actually ships: skip the slow, grounding-fragile generative route and use the LLM as a better encoder inside a two-tower you can already afford to serve. For large-scale retrieval teams, that means absorbing the LLM's semantic ability without ripping up existing retrieval infrastructure, and matching the baseline on 0.5% of the data says the pretrained knowledge transfers generously.
Distillation is a separate thread worth pulling: the ablation shows CE2TT distillation is the single biggest source of the student's performance. The heavy lifting is still a strong teacher pressing its knowledge into a light student; the LLM encoder provides a good base and distillation provides the supervision.
The authors note that the two-tower student's NDCG is less consistent than its R@10, so its ranking quality still trails the cross-encoder teacher. Larger configs (4B parameters, MoE, more latent reasoning steps) show headroom but already exceed the serving budget, so they are presented as future space rather than shipped results. Several tried designs (decoder-to-encoder conversion, L2-loss embedding matching, a latent step in the item tower) were cut for inconsistent gains, meaning the framework's components are not all equally battle-tested and the recipe still leans on empirical tuning.