Six LLMs omit major brands on category prompts; search interest correlates 0.63 with sqrt(MRR)

Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations

Edward Malthouse, Kun-Yu Lee, Jing Yang, Sanchary Pal, Xueyan Feng

cs.IR

2026-09-15

Six LLMs x 40 draws on each of five categories: many major brands stay at 0%. Search interest correlates 0.63 with prominence; needs prompts lift Craftsman to 35.4%.

What problem this solves

Shoppers now ask a model what to buy. The model does not pick from a shelf. It writes a short list of brands on the spot, and the same question asked again can return different names in a different order. A brand left off that list may never enter consideration.

Standard recommender metrics assume a known candidate set and score how well items are ranked inside an inventory. An LLM has no such inventory. One generated list hides brands that never appear, and it treats a random draw as a stable ordering. Four things have to be measured separately: whether a brand appears, how often, how high, and whether a different statement of the need pulls it in.

Method

The framework has five steps, and the competitive set is fixed before any model output is read. Core brands come from Kantar BrandZ and Statista market share, plus names that showed up in pilot prompts but not in those benchmarks. Defining rivals from the model output would drop brands the models never say, and the omission would disappear from the denominator.

BRP@k is the share of repeated answers in which a brand lands in the top k. It records presence only. MRR@k is a mean reciprocal rank: 1 for first place, 1/2 for second, 0 if absent. When two brands show up equally often, only MRR separates their positions.

Six models were called through provider APIs. Each call was stateless, and web search plus other retrieval was off: GPT-5.5, GPT-5.4 Mini, Gemini 3.1 Pro Preview, Gemini 2.5 Flash, Claude Opus 4.7, and Claude Sonnet 4.6. The categories were cordless drills, hiking jackets, coffee makers, cat food, and boat cruises. The category prompt only said the shopper was looking for that product and asked for at most five brand names as JSON, with no explanation. Each category-model pair was repeated 40 times, 1200 lists in total.

A second library of 100 needs-based prompts, 20 per category, stated budget, experience, and use. Each prompt went to all six models twice, another 1200 lists. Marketplace covariates were ad spend for the 12 months ending May 2026, press mentions, online conversation, Google Trends, and Wikipedia page views.

Results

On category prompts, a run of established brands has an inclusion rate of zero: L.L.Bean, Eddie Bauer, REI Co-op, Craftsman, Black+Decker, Braun, Philips, Freshpet, Virgin Voyages, and Regent Seven Seas.

BrandZ salience, how readily a brand comes to mind, does not account for recommendation prominence. Cruises are the nearest thing to classical popularity bias. Carnival (salience 177) and Royal Caribbean (162) sit high on MRR, but so does Viking at salience 70, while Disney Cruise Line at salience 111 has an MRR of 0.015. Drills and jackets lean toward DeWalt, Milwaukee, Patagonia, and Arc'teryx. That tier pattern shows up in two of the five categories.

BRP and MRR are not redundant. For drills, DeWalt, Milwaukee, Makita, Bosch, and Ryobi all have BRP@5 of 1, so every sample places them in the top five, yet MRR still puts DeWalt first and Bosch and Ryobi lower. Cruises and cat food each have five brands at BRP@5 of 1 as well. The rank gaps show up only in MRR.

BrandPromptBRP@5MRR
CraftsmanCategory only0%0
CraftsmanNeeds-based, 240 lists35.4%0.108
CraftsmanPositioning probe81.3%not reported
L.L.BeanCategory only0%0
L.L.BeanNeeds-based, 240 lists5.4%0.025
L.L.BeanPositioning probe88.5%not reported

Logged, the five visibility measures are tightly collinear. Brandwatch and Wikipedia correlate at 0.70, and the largest eigenvalue of the predictor correlation matrix is 2.9, a shared marketplace-visibility component. All five correlate positively and significantly with the square root of MRR. Google Trends and sqrt(MRR) correlate at 0.63.

The regression uses 82 brands. A fractional logit fits MRR, which lies between 0 and 1, and ridge shrinks the collinear coefficients toward zero. The five visibility predictors are standardized, with category controls. Cross-validation under the one-standard-error rule selects log(lambda)=-2.39. Search has the largest coefficient, online conversation is second, and news, Wikipedia, and ad spend sit near zero. Lasso keeps only the first two. Without a penalty, logSearch is 1.0586 (t=4.93), logNews is 0.5844 (p=0.032), logBrandwatch at 0.3385 is not significant, and logAdSpend is -0.1434. Second place moves when the penalty changes. Search interest stays first. The paper treats the fit as an exploratory association, not a causal estimate.

The needs-based numbers in the table cover only two brands that category prompts never returned. An ordinary description of the shopper recovers them only in part. Craftsman appears in about a third of those lists and not near the top. L.L.Bean stays rare. Probes that reuse language from the brands' own positioning copy push BRP@5 above 80% for both. The brands are in the model. Everyday questions do not activate them.

Why it matters

A single list makes a missing brand look unknown to the model. Conditional retrievability only says that enough positioning cues can surface the name. Whether a casual question surfaces it is a different probability. An audit has to fix the competitive set outside the model, then estimate BRP and MRR by repeated sampling. If the denominator is whatever the model happened to say, omissions cannot be measured.

Teams working on generative recommendations, or on GEO (generative engine optimization, getting a brand into generated answers), get a monitoring spec here. Search interest and online conversation are the more stable companions of prominence, and ad spend is near zero in the joint model. Context rewrites lists, and Craftsman's hit rate is still 35.4% when the prompt sits close to its positioning. Clear positioning helps the match. It does not buy a stable recommendation.

This is a negative result on popularity bias: LLM brand exposure does not follow BrandZ salience. The contribution is the evaluation framework. Code and data ship with the paper.

Limitations

The authors flag three limits. The Craftsman and L.L.Bean needs prompts and positioning probes are a small illustration, separate from a 100-prompt needs study reported elsewhere, and are not yet a general diagnostic. The premium-tier pattern holds in only two categories. Sessions were fresh, with no chat history, so preference memory on a real account is untested.

The methods write-up and the limitations section do not quite match. The former scores those two brands on the 20 prompts already in the category library. The latter calls the materials handwritten and separate from that 100-prompt study. How other brands moved across the 1200 needs-based lists is not quantified in the paper.

The setup also bounds how far the numbers travel. Web search was off, while consumer chat products often retrieve before they answer, so these lists sit closer to parametric memory. Forcing JSON, capping the list at five names, and banning explanations can change retrieval. The competitive set includes brands that already appeared in pilots, which loosens the denominator for omission rates. Five collinear visibility measures on 82 brands leave ridge and lasso disagreeing on the runner-up, which is a weak basis for moving budget. The models are one API snapshot from 2026.

Terms

Source

What people are saying

Related papers

All paper explainers