MIT's HugAgent benchmark finds AI digital twins struggle to predict you beyond interviews
新智元 · wechat · 2026-10-11
Researchers from MIT and collaborators present HugAgent (EMNLP 2026 Main Oral), a benchmark testing whether LLMs can build a digital twin of a specific person from interviews and predict their answers on unseen scenarios. The study recruited 54 real participants across three divisive topics—universal healthcare, public surveillance, and housing densification—capturing both belief states and belief updates under changed conditions (e.g., healthcare saving a family $3,000/year), for 1,742 evaluation items across GPT, Claude, Gemini, DeepSeek, Llama, and Qwen models.
Key results: models are fairly good at inferring current beliefs (best method 77.56%, GPT-4o 74.66%, vs. an 84.84% self-consistency ceiling), but notably worse at predicting answers under new conditions (best Claude Sonnet 4.5 at 68.61% vs. an 85.66% ceiling). Personal interviews matter—a lot (GPT-4o jumps from 39.83% to 63.11% on updates)—yet longer interviews do not reliably improve new-scenario prediction. The hardest part: judging what change is big enough to flip someone's stance, i.e., their change threshold.
The paper argues that reproducing known opinions is only the starting point; genuinely representing a person in unfamiliar situations remains the fundamental open challenge for human simulation.
More from AGI Musings
- Todd: putting your forecast numbers in writing is what really counts — ben_j_todd · 2026-10-23
- Humanities dissertations have collapsed, social sciences are next — with AI as one driver — gsiemens · 2026-10-23
- Greg Brockman: AI makes hacking 100x more profitable and easier, upending hacker economics — gabriel1 · 2026-10-23
- 'Nothing is real now on social media': viral demo shows AI-generated content everywhere — saibharadwaj · 2026-10-23
- Lilian Weng to Discuss Recursive Self-Improvement at SGLang Summit 2026 — BanghuaZ · 2026-10-23
- mmitchell_ai questions whether the AI safety vs security framing stems from EA — mmitchell_ai · 2026-10-23