AI eval researcher: even humans can't detect subtle AI patterns like distributional biases
alexisjross · x · 2026-10-06
Responding to a discussion, researcher ericzelikman explains his team invests heavily in building custom evals because humans themselves are poor at detecting subtle AI patterns, especially distributional biases.
He notes that for their user models the goal is to match the actual human distribution, not to fool people — an interesting stance on how to evaluate AI systems that simulate human behavior.
More from Models
- Why don't modern LLMs know time has passed between messages? — dumierhan · 2026-10-06
- Reflection AI's new text model reportedly pretrained on ~24T tokens, multimodal version expected — nagpalchirag · 2026-10-06
- Early User Reports Anthropic's Opus 5.5 Fills Its Context Window Quickly — rickasaurus · 2026-10-06
- Viral Claude vs GPT Charts Mislead: Claude's "5x Value" Is Mostly Just Higher API Pricing — jdjohnson · 2026-10-06
- Rumor: Zhipu's next open source release GLM 5.5 may beat Claude Opus — bindureddy · 2026-10-06
- Liquid AI's d1 vision decision model matches GPT-6.1 Sol on 4 of 6 tasks at 19x-200x lower cost — JosephJacks_ · 2026-10-06