Programming benchmarks overrepresent Python, Go and Rust compared with real usage
zainhas · x · 2026-07-24
Programming benchmarks are badly skewed toward Python, Go, and Rust
The chart compares benchmark task share with real-world developer usage across programming languages.
What it shows
- Benchmark coverage is heavily concentrated in Python, Go, and Rust.
- Real-world usage is much broader, with languages like JavaScript, TypeScript, Java, C#, C++, PHP, and others appearing more prominently in developer practice than in benchmark suites.
- The blue bars represent benchmark task share, while the orange diamonds represent developer usage from the Stack Overflow Developer Survey 2024.
The main point is that benchmark design does not reflect actual language distribution in professional software work, which can distort evaluation outcomes.
More from Research
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27
- Agentic Data Science in Practice: Agents Write Code but Answer Wrong Questions — hugobowne · 2026-07-27
- A concise canon of foundational papers in ML, systems, NLP, speech, and audio — deliprao · 2026-07-27
- TechCrunch says brain-wave signals could be the next unlock for physical AI training — TechCrunch AI · 2026-07-27