Researchers debate credibility of coding benchmarks as pass@k can be gamed

HF researcher eliebakouch argued perplexity alone is a weak metric for code models and pass@k matters more, while startup exec bneyshabur admitted pass@k scores can be inflated with synthetic data, sparking debate on the credibility of coding benchmarks.

2026-10-24 ~ 2026-10-24 · 4 related posts

• Researcher: perplexity evals favor code-only models; pass@k would be far more informative — eliebakouch · 2026-10-24
• Researchers spar over code perplexity evals as Kimi-K3 tops all 10 codebases — eliebakouch · 2026-10-24
• Startup exec admits pass@k coding benchmarks can be gamed with synthetic data — bneyshabur · 2026-10-24
• pass@k can be gamed by synthetic data: researchers debate post-training benchmark reliability — eliebakouch · 2026-10-24