Programming benchmarks overrepresent Python, Go and Rust compared with real usage
zainhas · x · 2026-07-24
Programming benchmarks are badly skewed toward Python, Go, and Rust
The chart compares benchmark task share with real-world developer usage across programming languages.
What it shows
- Benchmark coverage is heavily concentrated in Python, Go, and Rust.
- Real-world usage is much broader, with languages like JavaScript, TypeScript, Java, C#, C++, PHP, and others appearing more prominently in developer practice than in benchmark suites.
- The blue bars represent benchmark task share, while the orange diamonds represent developer usage from the Stack Overflow Developer Survey 2024.
The main point is that benchmark design does not reflect actual language distribution in professional software work, which can distort evaluation outcomes.
More from Research
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11
- Bug Hunt Bench ranks frontier coding models on 105 planted real-repo bugs — PawelHuryn · 2026-09-11
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Johns Hopkins Launches Full-Stack Hands-on Robot Learning Class with SO-101 Arm Kits — _krishna_murthy · 2026-09-11