Colgate paper: LLM roleplay hits 90% of human test-retest reliability in purchase-intent research
2C_ornot2C · x · 2026-11-21
Colgate-Palmolive and PyMC Labs published a paper introducing Semantic Similarity Rating (SSR): instead of asking LLMs for numeric ratings (which yield narrow, skewed distributions), they have the model roleplay a demographic persona, write raw purchase-intent thoughts, then map those texts to a 5-point Likert scale via embedding similarity to anchor statements.
Key results:
- Tested on 57 real personal-care surveys from a leading consumer company (9,300 human responses), SSR achieves 90% of human test-retest reliability with realistic distributions (KS similarity > 0.85).
- Synthetic respondents also produce qualitative feedback explaining their ratings, preserving interpretability.
The authors argue the flaw lies in elicitation methods, not the models themselves, positioning SSR as a scalable alternative to traditional consumer panels.
More from Research
- Ramez Naam extrapolation: AI a decade away from completing 98% of hour-long human tasks — GaryMarcus · 2026-11-21
- Forecast-reality gap in AI task lengths hits 165x and widens ~30x over 12 months — GaryMarcus · 2026-11-21
- AI 2027 forecast overestimates OpenAI's real-world task lengths by 111x, analysis shows — GaryMarcus · 2026-11-21
- Riemann Hypothesis proof gets full formal verification, experts call it 'very human math' — Kyrannio · 2026-11-21
- Claims surface: Navier-Stokes and Riemann proofs via LEAN, by OpenAI and Claude — BasketVolleyball11 · 2026-11-21
- Hassabis predicted 2 years for AI to crack a Millennium Prize problem; 6 months later, both Riemann and Navier–Stokes have proposed solutions — Hesamation · 2026-11-21