Colgate paper: LLM roleplay hits 90% of human test-retest reliability in purchase-intent research

2C_ornot2C · x · 2026-11-21

Colgate-Palmolive and PyMC Labs published a paper introducing Semantic Similarity Rating (SSR): instead of asking LLMs for numeric ratings (which yield narrow, skewed distributions), they have the model roleplay a demographic persona, write raw purchase-intent thoughts, then map those texts to a 5-point Likert scale via embedding similarity to anchor statements.

Key results:

The authors argue the flaw lies in elicitation methods, not the models themselves, positioning SSR as a scalable alternative to traditional consumer panels.

Original post →

More from Research

Research channel →