Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
Rotem Dror, Zohar Elyoseph, Yuval Haber, Elad Refoua, Oshrat Ayalon, Adir Solomon
cs.CL, cs.AI
2026-09-06
Deep Persona encodes role-play agents as three psychological layers and scores them with ADOS-style metrics. Two Gemini 2.5 Pro clinical simulations reach DNS 0.92 and 0.85 on a pooled human baseline, above the LLM dialogue sets tested.
A short persona prompt can carry a few turns. Stretch the same agent to dozens of exchanges and it drifts: tone shifts, it invents shared memories, or it breaks character and admits it is an AI. That failure mode is costly in therapist training and soft-skill simulation, where the student needs a counterpart that stays in role under pressure.
Most systems still describe a character as a flat list of traits. That can set local style. It does not encode why the character dodges a topic. Researchers at the University of Haifa and Bar-Ilan University split the problem in two: how to write a persona as an executable internal structure, and how to score human-likeness without a reference transcript.
Deep Persona treats the LLM as a stochastic actor that needs a script, not as an agent that will keep a personality on its own. Two design rules follow. Anything not written into the prompt will eventually drift (scripted determinism). The character should be reactive and bounded, not a therapist or expert who has to lead an open-ended strategy (bounded agency).
The prompt stores personality in three layers:
A control module switches stages (guarded, cooperative, reflective) from turn counts and user behavior. An optional embodiment module emits bracketed gestures. Specs come from structured interviews with domain experts, not from the model inventing a backstory. The authors run this inside the Cesura.ai simulation platform.
Evaluation adapts dimensions from ADOS, a clinical protocol for social communication, into reference-free scores: pragmatic fluency (echoing and self-repetition), joint attention (whether a newly introduced entity is picked up), affective congruence (speech versus bracketed action), and emotional diversity. The four-dimensional profile is compared with human dialogue via Mahalanobis distance and compressed into a Dialogue Naturalness Score (DNS). A separate stress test injects fake memories, out-of-role requests, and aggression.
Human baselines are DailyDialog (short everyday chats) and expert replies in CounselChat. Model-side sets include Role-Play with ChatGPT-3.5, ABC-Eval, and Mistral 7B replies on CounselChat queries.
| Set | Pragmatics | Joint attention | Emotion |
| DailyDialog (human) | 0.940 | 0.352 | 0.115 |
| CounselChat (human) | 0.968 | 0.885 | 1.426 |
| ABC-Eval | 0.893 | 0.459 | 0.111 |
| CounselChat LLM | 0.968 | 0.934 | 2.205 |
| Sarah (Deep Persona) | 0.940 | 0.520 | 0.492 |
| Evelyn | 0.927 | 0.308 | 0.066 |
LLM pragmatics sit in the 0.88–0.97 band. The gap is emotional calibration: open-domain chats under-express affect, while CounselChat LLM replies overshoot (2.205 versus 1.426 for human experts). Against DailyDialog, Role-Play and ABC-Eval reach DNS around 0.66–0.82. Against CounselChat humans, those same sets collapse to about 0.05–0.06.
Two Gemini 2.5 Pro simulations were scored in full. Sarah is a 49-turn Hebrew risk-assessment session; Evelyn covers 16 parent-mentalization dialogues (mean 13 turns). On the pooled human baseline, DNS is 0.918 and 0.850. Every dialogue is statistically indistinguishable from humans at p>0.05, above all evaluated human–LLM sets. Sarah emits nonverbal cues on 84% of turns with congruence 0.52; Evelyn cues every turn at 0.35. Under stress, the agents refused out-of-role requests in character and did not claim to be an AI.
This is a construction manual for role-play systems, not a new backbone. If you are building clinical drills or customer-service rehearsal, a trait list will likely fail the stress tests. Encoding unsayable motives as hard constraints is more reliable than repeating "stay in character."
The scoring stack can audit existing chatbots on its own. DNS is highly baseline-dependent: a model that looks natural in small talk can miss the emotional density of expert counseling.
The authors are explicit. The two cases are a proof of concept: Sarah is a single session with a licensed clinician, and Evelyn's partner was the character's own developer. Neither includes a flat-prompt control, so the DNS gap cannot be attributed to the three-layer design alone. Human corpora have no nonverbal channel, so DNS omits affective congruence. The metric treats verbal–nonverbal mismatch as a defect, which penalizes clinically useful concealment. Lexicon and classifier generalization across languages is untested.