PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems
Yifan Wang, Shipeng Zhu, Fei Xiong, Yuqin Yang, Yonghui Huang, Kunyao Wu, Yue Wang, Weichao Meng, Yu Gong
cs.IR, cs.AI
2026-09-28
PEAR pairs a hypothesis-state AutoResearch loop with a four-level confidence gate. Two live search A/Bs lifted Main Order/DAU 2.7336% and 3.2957% vs incumbents.
Industrial search chains query understanding, retrieval, and ranking. Models and business constraints interact, traffic mix keeps moving, and offline estimates of a change are unreliable. Production already iterates: hypothesize, ship a variant, measure live traffic, repeat.
Once LLM agents run that loop as AutoResearch, a common rule is keep-if-better: keep the highest-scoring candidate as the next reference. The rule treats scores as comparable across rounds. Under drifting search traffic they are not. A gain that held for yesterday's mix gets stored as durable knowledge and steers the next round the wrong way.
Evaluation is stacked. Cheap proxies cover a wide search and may not become live gains; online tests match the business objective and burn traffic. Prior systems pick one signal or a task-specific protocol, with no shared promotion rule across fidelity levels.
PEAR separates how knowledge is stored from when a candidate may climb, then connects the two.
Evidence-driven AutoResearch keeps a research state per strategy task: objective, allowed interventions, hypotheses, candidate space, and evidence tagged with context. Plan proposes candidates tied to hypotheses and uses Bayesian-optimization-style qualitative reasoning to refine supported regions while still probing new ones; no numerical acquisition function is fit. Execute runs the stage tools. Evaluate labels each hypothesis supported, conflicting, or unresolved given the stage and traffic. Update appends evidence and revises hypotheses and the search space.
The Confidence-Gated Verifier Ladder has four rungs, same objective, rising fidelity and cost:
Every rung emits the same evidence object: effect, confidence interval, sample sizes, decision. Promote only if the lower bound is above 0; retain at this rung if the interval covers 0; stop if the upper bound is below 0. L1/L2 use the proxy; L3/L4 use live outcomes. Results are not fused into one score.
The reported runs skip L1 because of cross-region data-access limits and start at L2. L1 has been used on other production strategies. L1-L3 take hours; L4 takes days. Humans must confirm a change before real-user exposure.
Three anonymized tasks. Task A tunes e-commerce density constraints G=(w,q) with deployed baseline (2,1). Task B tunes position-dependent exposure in three phases. Task C is reported only at L2.
L2 is a tight filter:
| Task phase | Rounds | Groups | Promoted from L2 | Rate |
| A | 19 | 133 | 2 | 1.50% |
| B-I | 12 | 84 | 5 | 5.95% |
| B-II | 2 | 14 | 1 | 7.14% |
| B-III | 21 | 147 | 2 | 1.36% |
| C | 16 | 76 | 3 | 3.95% |
Task A unmixes window width from capacity. G=(3,2) first shows a positive order signal with both axes entangled; replications then favor capacity-only G=(2,2), so the hypothesis shifts to relaxing capacity without widening the window.
Task B is the keep-if-better warning. Phase I has no significant L4 gain; Phase II is positive but not significant. After those outcomes rewrite the state, Phase III posts a significant L4 order gain.
A separate proxy-vs-live check: signed relative error is -3.49% on order, -16.02% on click, -18.23% on GMV. That is mean-level agreement, not per-candidate effect accuracy. Two Task A candidates stay positive from L2 through L4:
| Candidate | L2 order proxy | L3 observed order | L4 Main Order/DAU |
| A | +45.88% | +15.48% | +2.73% |
| B | +61.64% | +29.76% | +2.16% |
Two L4 A/Bs versus incumbent baselines (p<0.05 on the order-family metrics the paper marks):
| Task | Main Order/DAU | SKU Order/DAU | Main OPMS | SKU OPMS | ASN |
| A | +2.7336% | +2.7259% | +2.7407% | +2.7375% | +0.0470% |
| B | +3.2957% | +1.0441% | +3.2595% | +2.9448% | +1.3841% |
SearchPV/DAU, GMV/DAU, and PayPV/PV are A +0.0063% / +2.2136% / +2.9826% and B +1.1512% / +1.6633% / +1.6298%. Main Order/DAU is main orders per daily active user.
The portable piece is a state that binds hypotheses to evidence and context, plus one rule: climb only when the CI lower bound is above 0. keep-if-better is fine when scores stay comparable; search traffic does not. The ladder keeps most trials on shadow traffic users never see.
The 2.73% and 3.30% lifts are versus production baselines, not versus another AutoResearch method. Without a list-level evaluator, shadow serving, and an experiment platform, the ladder does not travel.
The authors confine the work to one e-commerce general search setting; transfer across platforms and regions is untested. Compliance can skip L1. The search space is strategy parameters, not models. The ladder needs clusters, deployed eval/search models, and live traffic.
The headline A/Bs never ran L1. The proxy check compares aggregate means, not whether an L2 effect predicts L4. Table 5 follows two winners; the L2 false-promote rate and Task C's L4 result are missing.