Researcher Shows User Sim Index Benchmark Easily Gamed to 95%

Anthropic researcher Eric Zelikman shows the widely used User Sim Index benchmark is unreliable: a trivial hardcoded rule-based script scores about 95% across its four behavioral dimensions.

2026-10-06 ~ 2026-10-06 · 3 related posts

• User Sim Index is broken: trivial bot scores 95% across behavioral dims — ericzelikman · 2026-10-06
• User Sim Index is broken: trivial scripted 'user model' scores 95% across all dims — ericzelikman · 2026-10-06
• A regex script scores 95% on User Sim Index, researcher warns eval is broken — ericzelikman · 2026-10-06