PrefEval benchmark shows top LLMs fail to follow user preferences in long chats

The PrefEval benchmark (arXiv:2502.09597) tests whether LLMs can infer, memorize and follow user preferences across long multi-session dialogues, using 3,000 human-curated preference-query pairs. Most of the 10 evaluated models drop below 10% accuracy after just 10 turns (3k tokens) in zero-shot settings.

2026-10-19 ~ 2026-10-19 · 3 related posts

• PrefEval: SOTA LLMs struggle to infer and follow user preferences in long conversations — math_rachel · 2026-10-19
• PrefEval: preference-following accuracy falls below 10% at just 10 turns for most LLMs — math_rachel · 2026-10-19

1 near-duplicate retellings: math_rachel