GPT-2 Small-World Topology Likely an Architectural Artifact
its_vayishu · x · 2026-07-15
Upon reproduction, the author found that a trained GPT-2 does resemble a small-world network by this metric.
However, the untrained control group yielded similar results (σ ≈ 4.6, compared to ≈ 5 post-training).
They repeated the test across 5 sparsity levels, and the gap remained negligible, suggesting this is a result of the architecture itself rather than something "learned" during training.
Conclusions:
- top-k attention graphs might inherently exhibit small-world characteristics simply by construction
- it's invalid to claim that "GPT-2 secretly developed brain-like topology"
- the author has uploaded the complete notebook to Kaggle, welcoming verification and feedback
Related event: GPT-2 Attention Shows Small-World Properties(2 posts)→
More from Research
- Navier-Stokes, Riemann, P vs NP: what this week's math buzzwords mean for you — koltregaskes · 2026-09-11
- Fruit fly connectome LLM weights land on Hugging Face, transformers-compatible — ngxson · 2026-09-11
- Fruit fly brain as an LLM: connectome-driven language model demo goes live — ngxson · 2026-09-11
- Harry Collins: LLMs can't do frontier science because they can't invent new language — whoamisri · 2026-09-11
- The Waymo effect: how AI is quietly making research less collaborative — JohnHammersley · 2026-09-11
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11