GPT-2 Small-World Topology Likely an Architectural Artifact
its_vayishu · x · 2026-07-15
Upon reproduction, the author found that a trained GPT-2 does resemble a small-world network by this metric.
However, the untrained control group yielded similar results (σ ≈ 4.6, compared to ≈ 5 post-training).
They repeated the test across 5 sparsity levels, and the gap remained negligible, suggesting this is a result of the architecture itself rather than something "learned" during training.
Conclusions:
- top-k attention graphs might inherently exhibit small-world characteristics simply by construction
- it's invalid to claim that "GPT-2 secretly developed brain-like topology"
- the author has uploaded the complete notebook to Kaggle, welcoming verification and feedback
Related event: GPT-2 Attention Shows Small-World Properties(2 posts)→
More from Research
- Stanford Team Introduces Gigatoken, the World's Fastest Tokenizer — StanfordAILab · 2026-07-22
- Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
- Reddit points to OpenAI’s ChatGPT Ads page — EcstaticAsparagus509 · 2026-07-22
- Open-source runtime lets each repo define its own AI code reviewer — ibabufrik · 2026-07-22
- DeepSWE: A New Benchmark for Evaluating AI Coding Agents on Real GitHub Issues — pmz · 2026-07-22
- A Rust space-economy sim runs hundreds of autonomous ships, built with Claude — kalcode · 2026-07-22