A simple test for memorization: delete creative samples and probe the vector space
sytelus · x · 2026-10-23
sytelus proposes a leakage-free way to test whether models truly generalize: remove a set of samples you find highly creative and unique, rebuild the vector space, and see if the deleted points can still be located — probing creativity vs. memorization.
Related event: Developer Proposes Method to Test if Models Memorize Training Data(2 posts)→
More from Models
- 'Wait' markers are inherited from training data, not invented by RL, HKUST study finds — huawei-noah · 2026-10-23
- Study: Claude Opus 4.8 beats human experts on every measure as peer reviewer of 124 Harvard papers — emollick · 2026-10-23
- AI power user: back in the groove with Claude, still wishes they'd used Perplexity instead — eschadiol · 2026-10-23
- OpenCode's free tokens are capped and temporary — a GTM incentive, not zero markup — AccBalanced · 2026-10-23
- Smaug-Nemotron-3.5 debuts as a top US open-source agentic coding LLM that runs on a single machine — bindureddy · 2026-10-23
- DeepSeek on RDNA2/3 + DDR4: 19 t/s peaks, 9 t/s average decode on 56GB VRAM — EmPips · 2026-10-23