Paper proves transformers forget prompts via power-law decay, and trained models forget faster
Shuyang Xiang · hf · 2026-10-20
Independent researcher Shuyang Xiang published A Conditional No-Hair Theorem for Transformers, borrowing an analogy from black-hole no-hair theorems to quantify how much of the prompt an autoregressive transformer retains after generating t tokens.
- With two different prompts followed by identical continuations, the distance between hidden states decays as a power law t^-α; α=1 matches the shrinking share of a fixed prompt under uniform attention, smaller α means longer memory
- Beyond the first layers, trained models forget faster (α≈0.55–0.7) than untrained ones with the same architecture (α≈0.2–0.5, decreasing with depth)
- Proves an O(P(log t)^(L-1)/t) bound for prompts of length P and depth L, holding for finite-depth pre-LN softmax transformers with full causal attention and bounded attention logits (e.g., rotary embeddings)
More from Research
- Open-source K-12 math knowledge graph: 674 concepts, 2,843 questions, mistakes traced back to primary school — aigclink · 2026-10-20
- SIGSPATIAL 2026: satellite VLMs recover 98.3% of travel-census accuracy; AlphaEarth wins cross-continental transfer by 33% — flosalim · 2026-10-20
- Gufo, an open-source Strix Halo inference engine, explains its hybrid model caching design — fedeizzo · 2026-10-20
- Metis Reasoning launches with The Age of Reasoning white paper on evidence-grounded AI reasoning — ThoreG · 2026-10-20
- RealityBench Benchmarks VLMs on Documenting Physical Work, Finds Weak Spatial Reasoning — relantic · 2026-10-20
- LLM masters undocumented compiler framework, raising the question: should DSLs be designed for machines? — GianniMariani · 2026-10-20