Therapist's experiment: swapping AI models preserves a companion better than removing memory (90% vs 27%)
Michaelarobards · reddit · 2026-10-17
A psychotherapist built an AI companion with persistent memory and ran blind-judged experiments across four Claude models, with and without memory.
Key findings:
- Same model twice: 92% sameness; different models with memory: 90%; different models without memory: 82%.
- Same model with vs. without memory: only 27–33% — removing memory disrupts the companion more than swapping the underlying model.
- Factual accuracy rose from 26–40% (no memory) to 78–83% (with memory); epistemic honesty stayed at 93–96% either way, driven mainly by instructions.
- Interpretation: instructions create a recognizable persona; accumulated memory gives it a biography.
Limitations acknowledged: single-day experiment, Claude-family models only, model-based judges, no human ratings yet. Next steps include human-blinded evaluation and separating user-memories from self-memories.
More from Research
- RECON paper: reusing REPA's projection head at inference makes DiT training 25x faster — udmrzn · 2026-10-17
- Princeton & UPenn's Recurrent Looped Transformer Carries State Across Tokens — TheTuringPost · 2026-10-17
- Zephon's TorchTitan integration resumes training across GPU counts with identical batches — josh_wills · 2026-10-17
- Mila Opens PhD Position on AI Evaluation from HCI and Technical Angles — windx0303 · 2026-10-17
- Nick Cammarata proposes multi-resolution, no-pruning attribution graphs to fix mech interp errors — inductionheads · 2026-10-17
- RUC open-sources EvoOntology: self-evolving ontology layer lifts data agents by up to 26.7 points — udmrzn · 2026-10-17