Learning User Simulators with Turing Rewards
Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim
cs.CL
2026-06-18
Turing-RL trains Qwen3-8B with a judge score for indistinguishability. Humans pick it over real users 57% of the time in chat; GPT-5 sounds too much like an assistant.
The default job for an LLM is still the assistant. A different job is growing: simulate a specific user, so agents have a sparring partner, personalization systems have a testbed, and computational social science has a cheap proxy. The hard part is the support of the distribution. One person, one context, many next utterances that would have been in character. Most training recipes still pin the signal to the one recorded reply, either by maximizing its log-probability or by scoring how close a generation is in content. The model learns to echo that sentence. It does not learn the user's response distribution.
The Turing Test already names the right target: a judge cannot tell the machine from the human. Researchers at MIT, Stanford, and MIT-IBM Watson AI Lab turn that target into an RL reward and call the recipe Turing-RL.
Each user gets a history block h that never overlaps the current session, plus a persona ρ induced from h by an auxiliary model. The default user encoding is u=(h, ρ). Given context x, the policy should emit a y that this person could have said.
Training has two stages. Qwen3-8B instruct first writes a chain-of-thought for each ground-truth reply, and LoRA SFT warms up on the trace plus [HUMAN]: y. GRPO then samples 4 candidates per example. The judge, Qwen3.5-397B-A17B, sees the user's history and two replies (the real one and the model's, order shuffled) and scores 1-7 on which is more likely written by that user, using local goal, motive, and style. 1 favors the real reply, 7 favors the model. The reward caps the score at 5 and rescales: (min(s,5)−1)/6. The cap is there because early runs learned to be "more human than the human," a clean case of reward hacking. A length penalty also docks replies that are much shorter or longer than the ground truth, with a heavier fine on the short side.
The policy is Qwen3-8B with thinking turned off, LoRA rank 64, alpha 32, 3 epochs. Two controls share the same SFT init and the same GRPO loop: Sim-RL uses HumanLM's content-similarity reward; Logprob-RL follows Gandhi et al. (2026) and uses the length-normalized log-probability of the ground truth given the model's own trace.
Two domains. PRISM multi-turn chat: 1,288 users, 128 held out, 880 turns. ConvoKit Reddit threads: 1,282 users, held-out subreddits r/tifu and r/worldnews, 267 examples. SFT users, GRPO users, and test users are disjoint, and history never leaks into the target.
Automatic eval switches the judge to Claude Sonnet 4.6 so the training judge is not grading its own students. Turing scores run 1-7, higher means harder to tell from the user. Turing-RL leads Sim-RL, SFT, and Logprob-RL in both domains, with a larger gap on chat: 4.31 on chat and 3.68 on Reddit under the default encoding. GPT-5 and Qwen3.5-397B, far larger models, barely beat the Qwen3-8B base on this metric. They write long, hedged, assistant-like replies. On content similarity, Turing-RL matches Sim-RL, which was trained to maximize that score, and both beat SFT. Indistinguishability does not trade away content overlap.
The human study is a forced-choice Turing test. 360 Prolific annotators, 600 binary judgments per condition, after a comprehension filter. Win rate above 0.5 means annotators picked the model over the real user.
| Method | Chat win rate | Reddit win rate |
| SFT-Init | 0.49 | 0.41 |
| Sim-RL | 0.50 | 0.52 |
| Turing-RL | 0.57 | 0.50 |
On chat, Turing-RL beats SFT (p=0.044) and Sim-RL (p=0.022) by paired permutation tests. On Reddit it beats SFT (p=0.0095) and ties Sim-RL (p=0.61). Reddit items are longer and slower to read; the authors treat the Reddit human numbers as a trend check. Ablating the user encoding (history, persona, or both) barely moves Turing scores. Persona is enough to sound like the person, not enough to recover the exact content. Reddit specificity leans on raw history.
If you train user simulators, score personalization, or need a sparring partner for agents, the usable takeaway is the reward, not the architecture. Stop pinning the objective to one ground-truth sentence. Optimize for a judge who cannot tell. An 8B model with that signal outruns GPT-5 on a chat Turing test. Code is public.
This is a better training signal on a small policy. Nobody has shown yet that assistants trained against these simulators get better.
The authors list three limits. Only open-ended chat and forums were tested, not task-oriented dialogue, negotiation, or collaborative problem-solving. Only Qwen3-8B was trained; the gap between rewards could shrink or grow at frontier scale. Training depends on an expensive judge, Qwen3.5-397B-A17B, and will absorb that judge's blind spots. Human eval is a partial check; disagreement patterns were not dissected.
The 0.57 chat win rate has a ±0.050 interval, hugging 0.5: significant, small. Absolute similarity is only a few percent, so "no sacrifice" is a ranking claim against SFT, not a claim that the model restates the user. The length penalty pins reply length to the ground truth, so part of the Turing gain may be "write this short," not style or motive. A model that can impersonate a person can also forge messages and scale social engineering. Training used public or consented research data, and the stated use is aggregate study, not reproducing identifiable people.