Four years in, NLP needs longitudinal measurement of how chatbots change people

Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactions

Nicole Mitchell, Dhruv Agarwal, Maty Bohacek, Remi Denton, Roma Patel

cs.AI

2026-08-04

A position paper from Google Research, Stanford, and Cornell argues NLP must pivot from single-session safety evals to longitudinal measurement, combining behavioral-science scales with long-horizon conversation data to track risks like dependence and cognitive deskilling, and wiring the results into alignment training.

What problem this solves

Four years after ChatGPT shipped, the long-horizon consequences of chatbot use are showing up as documented case studies: users who spent months in sustained conversation with a model and then took actions that harmed their real lives. This position paper from Google Research, Stanford, and Cornell names these longitudinal effects and sorts them into four groups: socio-affective (parasocial attachment, emotional dependence crowding out human relationships), cognitive (offloading reasoning and memory until the skills atrophy), epistemic (opinions and goals drifting under many small nudges), and clinical (mental-health harm and compulsive use).

The root problem is the timescale of evaluation. Existing safety benchmarks, red-teaming, and RLHF all target risks visible within a single interaction. Most of the risks above are diachronic; they accumulate across weeks or months of sessions. Recommender systems and GPS navigation produced similar long-horizon effects before. The authors' read is that personalization, cross-session memory, and anthropomorphism amplify those old risks to a new scale.

Method

This is an agenda paper, not an experiments paper. The proposal is three pieces of NLP infrastructure:

Metrics: adapt six families of validated behavioral-science scales (psychometric, psychosocial, cognitive, value, well-being, utility) into text-based measurements that estimate psychological constructs directly from a user's language, rather than relying on post-hoc questionnaires. The authors add caveats: computationalized scales have construct-validity problems and serve only as weak approximations, and using them to give models personality tests misses the point. The model is the intervention; what needs measuring is its effect on people.
Data: a systematic comparison of three collection frameworks. Randomized controlled trials (RCTs) establish causality but run short, suffer attrition, and change behavior when users know they are logged. Field studies have ecological validity and real populations, but the data is almost entirely owned by platforms. User simulations are scalable and forward-looking, but trajectories drift at long horizons and amplify training biases. The conclusion: the three must be combined, since each has a structural blind spot.
Modeling and alignment: track opinion drift with computational social science, model user belief updating with the Rational Speech Acts framework, and borrow dynamic-systems methods to find inflection points in behavioral time series. These trajectory-level metrics should become rewards or constraints in RL training, so models optimize for long-term user well-being instead of short-term engagement.

Results

No original experiments; the evidence synthesizes published studies. Representative numbers: a four-week RCT had participants discuss socioaffective topics with ChatGPT, with measurable changes in self-reported loneliness and socialization; a 21-day RCT with a companion chatbot measured anthropomorphism perceptions and associated risks; a 12-week study tracked intimacy and self-disclosure over time.

The paper's central illustration: a user mentioning depressive thoughts ten times in one session reads as curiosity; the same ten mentions spread across ten weekly sessions signal a high-risk pattern. The token volume is similar, and current models, which infer time from accumulated context, cannot tell the two apart. The paper reports no empirical comparison of any new method; that is outside its scope.

Why it matters

Cross-session memory and personalization are becoming standard product features. This roadmap effectively specifies the acceptance criteria for those features: not just "was this single interaction safe" but "what happened to the user after three months". For researchers, the scales and longitudinal designs already exist in psychology and HCI; the bottleneck is data access, not methodology. For product teams, it reads as a compliance and ethics checklist they will face sooner or later.

Limitations

The authors list three: naturally occurring longitudinal conversation data sits almost entirely inside private companies; the scale inventories carry a strong Western bias and will misread cross-cultural settings; and longitudinal monitoring is in tension with privacy, requiring federated logging, local differential privacy, and dynamic consent regimes. A tension the paper leaves unexamined: the parties best positioned to run longitudinal monitoring are the same platforms that profit from engagement, a conflict of interest no technical design dissolves. As an agenda document, every claim here awaits its first wave of empirical tests.

Terms

Longitudinal effects: harms that only appear, or grow worse, as interaction time horizons extend.
Diachronic: visible only along the time axis, as opposed to a synchronic single-point snapshot.
Cognitive offloading: delegating reasoning and memory to a tool, with skill atrophy from disuse.
RCT: randomized controlled trial, the behavioral-science gold standard for causal claims.
Rational Speech Acts (RSA): a recursive Bayesian model of how speakers and listeners reason about each other's mental states to coordinate meaning.
Attractor state: a dynamic-systems region that behavior converges toward; used here to describe users sliding into high-risk patterns.