ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker
Siqiao Xue, Shuxuan Liu, Ning Hu
cs.CL
2026-09-25
ZooWork trains 0.6B/4B/8B shopping rerankers on cross-family LLM labels. ShopRanker-8B hits 83.4% on 10,511 structured ShopRank-Bench pairs, vs Jina-m0 at 79.2%.
Open rerankers such as BGE-Reranker-v2-m3, Jina, and Qwen3-Reranker already look strong on BEIR and MS MARCO. E-commerce ranking asks a different question: given two products that are both on topic, which one the shopper actually wants. Product type, a stated budget, the intended recipient, and explicit exclusions can each veto a candidate.
Four gold-tier pairs make the failure concrete. For "shape sorter", all four open baselines pick an office mail sorter. For "jumpsuit under $100 boho", they all pick the $239 item. BM25, BGE-large, BGE-v2-m3, and Jina-m0 miss every pair. The pattern is systematic: on gold pairs whose query states a hard constraint, open baselines drop 14 to 20 points versus unconstrained queries, while ZooWork-ShopRanker stays flat. Budgets hurt most. BM25 falls to 47.9, below chance; the cross-encoders lose 26 to 34 points. A relevance ranker has no reason to prefer the cheaper of two equally relevant products.
Real search traffic supplies authentic queries and candidates. It does not supply clean pairwise labels. The bottleneck is supervision.
ZooWork-ShopRanker is a set of LoRA adapters on Qwen3-Reranker at 0.6B, 4B, and 8B. Base weights stay frozen. Scoring is the base recipe: one forward pass, the yes-minus-no logit gap, no generated text. In a controlled 0.6B comparison, LoRA at about 1% trainable parameters matches or slightly edges full fine-tuning.
Labels come from a panel of reasoning LLMs used as a preference oracle. Training uses Qwen3.5-122B and Gemma-4-31B only; a pair is kept when both families agree in both presentation orders. Evaluation adds DeepSeek-V4-Pro so the benchmark is not judged solely by the training panel. The protocol is constraint-first: check hard constraints, then weigh style, color, fit, and quality among survivors. Constraints are whatever the query states, not a global attribute ranking. A hand-designed priority hierarchy anti-correlates with judged preference.
The 8B flagship is aligned directly on about 4.4k judged pairs with a pairwise logistic loss (Bradley-Terry / RankNet) mixed with general retrieval replay. The 0.6B and 4B models are not aligned from their own bases. The aligned 8B scores about 135k query-document examples; students fit those soft scores with binary cross-entropy, then sharpen on judged pairs. Distillation beats aligning the same small base, and at feasible scale volume beats a stronger panel teacher.
On-policy mining was tried and stalled. After one round, development easy accuracy moved from 0.836 to 0.844. Further rounds dried up: alignment polarizes scores, the smallest-margin pairs become two-good or two-bad, judges tie, and gold yield falls from 26% to 6%. The frontier is judge-ambiguous, not just hard to mine.
ShopRank-Bench is built from Gensmo private search traffic: 10,511 preference pairs in structured and natural-language product text. Of 23,000 judged pairs, 45.7% survive. Gold (1,843) has three families committed and agreed; silver (4,445) two; bronze (4,223) one. Two diagnostic tracks, AHP (1,500) and explicit budget (1,302), are scored separately. Training and benchmark share no queries and no exact pairs.
Structured preference track, n=10,511:
| Method | Overall | Gold |
| ShopRanker-8B | 83.4 | 96.5 |
| ShopRanker-4B | 81.2 | 96.3 |
| Qwen3-Reranker-8B | 79.2 | 93.2 |
| Jina-m0 (2.4B) | 79.2 | 90.8 |
| ShopRanker-0.6B | 76.6 | 91.2 |
| BGE-v2-m3 (0.6B) | 73.3 | 84.9 |
| BM25 | 56.2 | 69.0 |
On natural-language renderings the 8B scores 81.8, the 4B 79.5, and Jina-m0 78.1. On the NL gold tier the 0.6B reaches 91.2, above Jina-m0 at 88.8. Every size beats its unaligned base. On structured text the distilled 0.6B is statistically indistinguishable from the unaligned 4B (minus 0.9 points, 95% CI [-1.9, +0.2]); on prose the 4B base still leads. Every system drops 20 to 26 points from gold to bronze, and bronze is 40% of the set, so the aggregate score is pulled by the weakest labels.
Rerankers are not the ceiling. Zero-shot reasoning LLMs GLM-5.3 and Qwen3.5-27B reach 91.9 and 92.1 overall on structured pairs, eight to nine points above the 8B, at 3.4 s and 11.8 s median per call and about $3.6 per thousand pairs.
On three MTEB reranking tasks the 4B and 8B average 0.799, a small lift over Qwen3-Reranker-8B at 0.793. Preference alignment does not trade away general reranking. After merging LoRA, serving cost matches the base. On one H200 the 0.6B does 433 docs/s at batch 32, 2.8 times the 4B, on 4.5 GB versus 11.2 GB. BGE-v2-m3 remains 4.6 times faster at batch 32 and uses about a third of the memory: decoder rerankers win when quality binds; the encoder still wins when throughput binds.
Diagnostics cut harder. On AHP, every open baseline sits below 50% and systematically inverts the intended priority; the aligned 8B just clears 54.3. The hierarchy is the wrong target: scoring seven attributes separately and linearly pooling them tops out at 69.5% against judge labels, while a preference-trained 0.6B hits 82.5% on the same development slice. On the budget track, general preference alignment overshoots toward cheap. A 0.6B specialist trained on the programmatic threshold rule reaches 94.7 overall.
E-commerce search has lacked an open, reproducible preference-alignment recipe for the last ranking stage. This paper turns "relevance is not shopping preference" into a measurable gap and releases three model sizes plus a dual-format benchmark. For practitioners the takeaways are concrete: alignment is free at inference; on structured product text a distilled 0.6B matches an unaligned 4B; a stated budget is better handled by a hard filter or targeted supervision than by general preference training. This is solid domain alignment, not a new architecture.
Labels come from LLM judges, not clicks or purchases. The bronze tier is largely Gemma-decided. 54.3% of hard adjacent pairs are unanimous ties or conflicts, which caps clean on-policy labels. Natural-language product text is model-rendered from attributes, not scraped copy, and can drop attributes. The mix leans apparel; queries are English- and US-centric. Training data is not released. AHP and budget together show that alignment shifts price bias; it does not install "honor the stated budget."
An eight-to-nine-point gap to reasoning LLMs remains. Distilling judge-level preference into a millisecond reranker is still open.