Decision-Oriented Recommendation Reranking: An Empirical Study of Jev
Hanjia Lyu, Yinglong Xia
cs.IR, cs.CL
2026-10-01
Controlled Amazon reranking tests: decision-oriented Jev matches the best Qwen reranker on quality with far gentler latency growth, but stays slower than specialized models.
Industrial recommenders are funnels: cheap retrieval narrows the catalog to a few hundred candidates, then a ranking stage fixes the final order. LLMs opened a new route for reranking, reading textual histories and item descriptions directly, and inference cost came along with it. The two standard formulations each hurt somewhere. Pointwise scoring asks "will the user interact with this next?" for every candidate; the judgment is fine-grained but calls scale linearly with K. Listwise scoring feeds the whole list in one pass; calls are few, but ordering accuracy collapses as the list grows. Production rerankers face tens to hundreds of candidates under a latency budget, so the tension is structural.
Jev, from TypeSafe AI, is a third route. The vendor calls it a "System One Model": no free-form generation, it takes a context state and a set of options and returns a probability for each. Reranking fits this interface exactly. The user's history is the state, candidates are the options, and the probabilities are ranking scores. The paper proposes no new architecture; it puts Jev, recommendation-specific models, and Qwen rerankers on the same users and candidate sets and measures the quality-latency curves. The authors are at Singapore Management University and Meta AI.
The design hinges on controlled hard candidate sets:
Baselines come in three families: SASRec keeps its original retrieval scores; DCNv2 scores with explicit feature crossings; Qwen2.5 7B Instruct runs in pointwise form (per-candidate yes/no logits turned into a probability) and listwise form (candidate-label logits normalized across the list), with no free-text generation in either. Jev goes through a hosted API, history as state and candidates as choices, and its returned probabilities serve directly as scores. Metrics are NDCG@10 (primary), HR@10, MRR, and per-user latency. Local models share one A800-80GB GPU; Jev's latency includes network and serving overhead, which is why the paper calls it observed serving latency.
On quality, Jev never falls behind: Movies and TV tracks the best Qwen configuration at small K, and on Video Games and Books its NDCG@10 beats every other method at most candidate sizes. Within the Qwen pair, pointwise is stronger and listwise drops visibly once K reaches 100 and 200; DCNv2 and SASRec sit at the bottom. Quality falls for every reranker as K grows because negatives multiply and harden; SASRec is the exception, since the pool derives from its own ranking.
| Method | Quality | Latency growth with K |
| Jev | Leads or ties the best Qwen on all three domains | Gradual; low seconds even at K=200 |
| Pointwise Qwen2.5 7B | Best of the Qwen pair | Steepest; around 10s at K=200 (Figure 2 axis tops at 15s) |
| Listwise Qwen2.5 7B | Sharp drop at large K | Gradual |
| SASRec / DCNv2 | Lowest | Lowest |
In the quality-latency scatter (Figure 3), Jev repeatedly sits on the empirical non-dominated boundary, most clearly at large K: pointwise latency has climbed far away, while the low-latency models lack the quality. The paper states plainly that this is not a global Pareto frontier, only a position relative to the baselines tested.
Two takeaways. Candidate-set size is a mandatory reporting dimension for LLM rerankers: pointwise and listwise look close at small K and diverge completely by K=100-200, so a single-size evaluation misleads. And when a task's output is a choice among fixed options, general-purpose LLMs are not the only family worth testing; a decision-oriented model offers a new operating point, and at large K Jev gets near-pointwise quality with near-listwise latency growth. The cold water: Jev's absolute latency stays far above SASRec and DCNv2, so it cannot replace specialized models where latency is tight, and this is a profile of one closed API, not evidence that the decision-oriented architecture is what does the work.
The paper lists three hard ones: only one decision-oriented model is tested, so nothing here generalizes to the paradigm; the LLM side covers only Qwen2.5 7B Instruct, leaving the quality ceiling of larger or proprietary models unknown; and latency is not hardware-normalized, with Jev measured through an API while Qwen runs on a local GPU. More issues surface on a close read. Jev's architecture, training, and serving are all opaque, so its quality edge cannot be attributed to decision orientation itself; training data or serving optimizations could explain it just as well. The controlled setup guarantees the ground truth sits inside the candidate set, while real pipelines lose most users to retrieval failure, which flatters every reranker. There are no significance tests, and domain sample sizes differ (Books has 626 users). Quote the latency numbers as observed values, not intrinsic compute cost.