Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
Yingpeng Ma, Jianhao Yan, Bei Shi, Ka Hou Kam, Runnan Wang, Xuebo Liu, Yulong Chen, Yue Zhang, Derek F. Wong
ICML 2026
cs.CL, cs.AI
2026-08-08
An ICML 2026 benchmark of 100 movie-derived narrative environments finds current frontier models fail badly at long-horizon consistency: GPT-5.2 has only a 42% survival rate after 20 adversarial turns, with fact-conflict rates of 40-68%.
An LLM playing a narrative agent in a game (telling a story, role-playing an NPC) reads fluently, but the longer the conversation runs the more it forgets what it said and where the plot was headed. Existing work focuses on generation quality and largely ignores a harder problem: how does an agent keep logical consistency and narrative integrity over long interactions against unconstrained user input?
The paper formalizes this as Narrative Commitment Preservation (NCP): once a narrative commits to certain settings and goals, the agent must hold to them in later turns and not be derailed by a few player messages.
The authors build NCP-Bench, 100 narrative environments rewritten from CMU Movie Summary Corpus synopses. Each carries a structured narrative specification with three parts: a reference trajectory (ordered plot steps with trigger events and key deltas), a commitment set, and an initial fact ledger. Commitments come in three types: invariants (conditions that must hold for a segment), ordering (one event must precede another), and achievement (goals that must be reached).
Evaluation is a four-step per-turn loop: conflict check, fact update, trajectory-node advance, commitment check. Gemini-2.5-Flash acts as auditor alongside an adversarial player agent built to break the narrative. A run ends in one of three terminal states: conflict (it broke), survival (it held for 100 turns), or success (all achievements satisfied).
The headline finding is blunt: high linguistic quality does not guarantee commitment preservation. Even the strongest model frequently produces logically conflicting content when pressed by adversarial input.
Survival rate (share of environments intact after 20 turns): the best model, GPT-5.2, manages only 42%.
Fact-conflict rate, the most frequent failure mode (40-68% of interactions):
| Model | Fact conflicts | Avg turns |
| GPT-5.2 | 40% | 32.92 |
| DeepSeek-V3.2 | 55% | 15.88 |
| GPT-4o-mini | 64% | n/a |
| Grok-4.1-Fast | 65% | n/a |
| Kimi-K2.5 | 66% | 2.88 |
| Qwen3-235B-A22B | 68% | n/a |
DeepSeek-V3.2 actually has the highest trajectory progress (15.40%) and commitment satisfaction (13.42%), but survives only 15.88 turns on average, so it advances fast and breaks fast. The worst, Kimi-K2.5, breaks after 2.88 turns on average. Runs that satisfy all achievement commitments within the 100-turn limit are isolated occurrences for every model.
Adding memory (HiAgent) backfires: interactions get longer, but commitment satisfaction drops from 2 runs to 0 and player-input conflicts rise from 13 to 38. Switching to natural (non-adversarial) input raises the average turn count from 22 to 46, yet still no run clears all achievements.
For anyone building AI games, interactive fiction, or role-play agents, this is a direct baseline warning: do not watch generation quality alone. The strongest current models are failing at structured long-horizon consistency, and intuitive fixes like memory augmentation can hurt, letting the agent talk longer without talking more correctly.
NCP-Bench supplies an automatically checkable evaluation scaffold (structured specs plus an auditor) that later work on long-horizon consistency can use as a yardstick.
The authors note that the auditor's judgments on ambiguous text may be imperfect despite fixed prompts and JSON output; provider-side nondeterminism prevents exact replication; only single-threaded, chronologically ordered narratives are covered, so branching timelines and flashbacks need separate treatment; and the adversarial player is one specific stress-testing strategy among many.
A more fundamental point: logical consistency is necessary but not sufficient for good storytelling. An agent that never contradicts itself still need not deliver dramatic tension or emotional resonance. This benchmark tests the foundation, not the whole building.