GPT-2 Small-World Topology Likely an Architectural Artifact
its_vayishu · x · 2026-07-15
Upon reproduction, the author found that a trained GPT-2 does resemble a small-world network by this metric.
However, the untrained control group yielded similar results (σ ≈ 4.6, compared to ≈ 5 post-training).
They repeated the test across 5 sparsity levels, and the gap remained negligible, suggesting this is a result of the architecture itself rather than something "learned" during training.
Conclusions:
- top-k attention graphs might inherently exhibit small-world characteristics simply by construction
- it's invalid to claim that "GPT-2 secretly developed brain-like topology"
- the author has uploaded the complete notebook to Kaggle, welcoming verification and feedback
Related event: GPT-2 Attention Shows Small-World Properties(2 posts)→
More from Research
- TMLR paper: enforced thinking doesn't reliably make LLM agents disclose key info to users — SharonYixuanLi · 2026-10-27
- Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27
- How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27