Programming benchmarks overrepresent Python, Go and Rust compared with real usage
zainhas · x · 2026-07-24
Programming benchmarks are badly skewed toward Python, Go, and Rust
The chart compares benchmark task share with real-world developer usage across programming languages.
What it shows
- Benchmark coverage is heavily concentrated in Python, Go, and Rust.
- Real-world usage is much broader, with languages like JavaScript, TypeScript, Java, C#, C++, PHP, and others appearing more prominently in developer practice than in benchmark suites.
- The blue bars represent benchmark task share, while the orange diamonds represent developer usage from the Stack Overflow Developer Survey 2024.
The main point is that benchmark design does not reflect actual language distribution in professional software work, which can distort evaluation outcomes.
More from Research
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26
- Surprising find: Alzheimer's-linked amyloid fibrils deliver RNA to sperm for embryo translation — EricTopol · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26