Identity Loss and Seamless Recovery in LLMs at High Temperatures
cephaloform · x · 2026-07-30
A developer observed that under high-temperature generation, models (like Mistral and Llama) sometimes briefly lose their identity at the beginning of a message but then seamlessly recover and integrate the preset identity (like Claude) later. This might be due to RL training masking user replies from the loss function, causing the model to rely on base user modeling from mid-training, with the RL delta pulling it slightly out of distribution (OOD).
Related event: Claude Base Model's Anomalous Output Is User Modeling, Not Inner Voice(5 posts)→
More from Models
- Google restricts Gemini Pro access for Google Plus subscribers, matching free tier — EmergencyLess8861 · 2026-10-16
- Meta open-sources Muse Spark: runs on two Mac Studios, license bans passing output off as human-written — mawcopolow · 2026-10-16
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16