LLM Memorization: The 3.6 Bits Per Parameter Threshold
gabriberton · x · 2026-07-03
The paper "How much do language models memorize?" reveals that LLMs continuously memorize training data until saturation occurs. This saturation point is reached at 3.6 bits per parameter; beyond this, the model begins to compress → generalize → exhibit double descent → and experience grokking. This establishes a quantitative threshold for the transition from memorization to generalization.
More from Research
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26