Paper: How Much Do LLMs Remember? ~3.6 Bits Per Param
pmttyji · reddit · 2026-07-07
A new paper proposes a method to estimate a model's memory of individual data points to measure its capacity. The authors formally decompose memory into "incidental memory" (information specific to a dataset) and "generalization" (information about the true data generation process). By completely eliminating generalization, the total memory yields the capacity estimate: GPT-style models store approximately 3.6 bits per parameter.
When training on gradually increasing datasets, models first exhaust their capacity for memorization before entering the grokking phase, where incidental memory decreases and generalization increases. The researchers trained hundreds of transformers ranging from 500,000 to 1.5 billion parameters, deriving a series of scaling laws related to capacity, data scale, and membership inference.
Related event: ICML Paper: LLMs Store 3.6 Bits of Memory Per Parameter(2 posts)→
More from Research
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22
- A Reddit user proposes a consistency LoRA to keep anime and game scenes visually stable — ThirdWorldBoy21 · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- BlackboxNLP 2026 is recruiting extra reviewers after a high submission volume — hanjie_chen · 2026-07-22
- AWS shows self-distilled reasoning can preserve math and coding skills during SFT — AWS ML Blog · 2026-07-22
- UI2App shows screenshot fidelity still lags real interaction recovery — Grace Man Chen · 2026-07-22