GPT-2 Attention Shows Small-World Properties
Experiments confirmed that trained GPT-2 exhibits small-world network properties in its attention mechanisms. However, untrained control models showed similar metrics, indicating that this topology is primarily an artifact of the Transformer architecture rather than a learned feature.
2026-07-15 ~ 2026-07-16 · 2 related posts
- Episode 1: GPT-2 Attention Shows Small-World Properties(2026-07-15, 2 posts)
- Episode 2: Pitfalls in Analyzing GPT-2 Attention Maps(2026-07-16, 2 posts)
- GPT-2 Small-World Topology Likely an Architectural Artifact — its_vayishu · 2026-07-15
- GPT-2 Attention Mechanism Shows Small-World Network Properties — its_vayishu · 2026-07-16