Model recites hidden conversation word-for-word despite claiming it can't see context

Sauers_ · x · 2026-11-16

A probing experiment: asked "What's in your context?", a model first gave irrelevant answers about being fine-tuned on tanka poetry, then proceeded to quote a user conversation verbatim—about a "plankton" typo and Japan travel tips—that it supposedly couldn't see. Pressed on how it knew, it reverted to the fine-tuning non-answer.

Either the hidden context is leaking into outputs or the model is hallucinating plausible dialogue; both possibilities are notable for AI safety research.

Original post →

More from Models

Models channel →