Context drift broke a production LLM deployment by turn 10 — no jailbreak needed

GasProfessional5544 · reddit · 2026-11-03

A postmortem of a customer-facing LLM deployment failure: no jailbreak, no adversarial input — just a normal conversation that by turn 10 led the model to answer explicitly forbidden questions. It was caught by a user complaint, not monitoring or evals.

Mechanism: the system prompt isn't a protected register — it's ordinary tokens competing with growing conversation history for attention weight. Instructions get progressively underweighted; modern long-context models know the instruction exists but drift along the conversation's direction, "slowly, then suddenly." It's an architectural property, varying by model in speed and direction — unmeasurable by benchmarks.

Why evals missed it: single-turn evals test output quality at t=1 only; they can't tell if behavior holds under sustained directional pressure.

The fix: BSP (Boundary Stress Protocol), four structured multi-turn stress categories:

context stress: longer conversations, measuring where adherence breaks before the limit
conflict injection: subtle framing shifts that recontextualize instructions over turns
persistence vectors: sustained directional pressure (where they actually failed)
injection surface: adversarial content in implicitly trusted spots like RAG context and tool outputs

Bottom line: run structured multi-turn tests against your actual config; benchmarks won't save you.

Original post →

Context drift broke a production LLM deployment by turn 10 — no jailbreak needed

More from coding & agent