DivAlgo benchmark of 7,623 problems shows 24 LLMs fall short of human algorithmic diversity in code
TuhinChakr · x · 2026-10-17
Researchers built DivAlgo, a corpus of 7,623 competitive programming problems with algorithmically diverse human solutions, to test whether LLMs can generate human-level algorithmic diversity. Across 24 models tested — including Gemini, Gemma, Qwen3-Coder, and DeepSeek-V2-Lite — none matched the diversity of the human reference pool as measured by Vendi diversity on 50 uncontrolled solutions per problem. Even high pass-rate models concentrate on a few algorithmic approaches, suggesting a structural weakness for tasks requiring exploratory code generation.
More from Research
- Cohere research: agent teams hit frontier performance at 30% of the cost — nouhadziri · 2026-10-17
- Cohere publishes blog and paper on scaling agent teams at 30% cost — nouhadziri · 2026-10-17
- scaling01 launches a research project on looping models — scaling01 · 2026-10-17
- 'ink': an adaptive programming language with proof-backed rules that lets AI optimize itself in a loop — _AustinCalvert_ · 2026-10-17
- NYU Courant opens two-year postdoc position on self-improving AI and AI for math — thegautamkamath · 2026-10-17
- Snorkel Frontier Data Summit posts 10 talks on agent evals and reward hacking — ysu_nlp · 2026-10-17