CAS and Kuaishou's OmniBehavior benchmark: top models score just 44.55 at simulating real users
jiqizhixin · x · 2026-10-31
Researchers at the Chinese Academy of Sciences' Institute of Software and Kuaishou introduce OmniBehavior, accepted to NeurIPS 2026, showing user simulation remains far from solved: the strongest evaluated model scores only 44.55 on the composite metric.
The benchmark preserves long, connected behavior histories instead of isolated clicks: 2.12 million interactions from 200 users over three months, spanning 22 actions across 5 scenarios. Models get a user profile, historical traces and current context, then predict behaviors from binary actions to viewing duration and customer-service replies.
A recurring finding: models systematically overestimate positive engagement, soften negative language, and flatten differences between individuals.
More from Research
- Autoencoder experiments show bias-free activations yield monotonic representation evolution — ZimingLiu11 · 2026-10-31
- Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31
- Biology Isn't Just a Data Problem: The lncRNA ANRIL Breaks Every Model You'd Write — iskander · 2026-10-31
- KAN boosts TabPFN scaling forecasts: OOD error cut up to 80% on StepLaw tasks — ZimingLiu11 · 2026-10-31
- Pre-registered training-curve forecast hits: TabPFN covers 14/14 later checkpoints — ZimingLiu11 · 2026-10-31
- KAN author argues training-curve prediction fits tabular foundation models like TabPFN, not LLMs — ZimingLiu11 · 2026-10-31