WeightWatcher K-matrix power-law exponent tracks LLM memorization with ρ ≈ -0.90
rickasaurus · x · 2026-09-24
A preliminary analysis applies WeightWatcher to study how LLMs memorize random inputs: when part of the training data is deliberately corrupted with random labels, the model eventually memorizes them and clean test accuracy degrades.
Monitoring weight spectra throughout training reveals a strong, layer-specific signal:
- The clearest effect appears in the attention K matrix
- As random-label memorization grows, the power-law exponent α falls toward and then below the critical value α = 2
- The relationship is highly reproducible across independent seeds, with mean rank correlation of roughly ρ ≈ -0.90 between K α and memorization
- The effect survives WeightWatcher's fixfingers correction, ruling out artifacts from a few anomalous eigenvalues
The α exponent could serve as a real-time in-training monitor for memorization.
More from Research
- Epoch AI audit finds 46% of sampled Humanity's Last Exam questions are defective — geoffwolfe · 2026-09-24
- Semantic operators: LLM data processing at scale needs full-stack rethink — CShorten30 · 2026-09-24
- Jev processes 100k rows for $2.50 in under 60s, demo now public — CShorten30 · 2026-09-24
- CLM-8B hits SOTA 81.6% on DeepSWE with light finetuning, up to 9x faster inference — anshulkundaje · 2026-09-24
- SpeakerMem-R1 Tops EverMemBench at 62.33% with Dual-Track Multi-Party Dialogue Memory — zju · 2026-09-24
- CMU's WhatWorkedBench Measures How Well AI Research Agents Understand Their Experiments — CarnegieMellonU · 2026-09-24