Startup exec admits pass@k coding benchmarks can be gamed with synthetic data

bneyshabur · x · 2026-10-24

Responding to benchmark criticism, a startup model-company exec (bneyshabur) candidly admits that pass@k on coding benchmarks can be gamed via synthetic data, and even post-training benchmarks are often overoptimized.

Key points:

Nothing is established until people actually use the model; evals are just extra data points.
A model trained only on code can beat larger models on perplexity, but real-world coding benchmarks are less certain.
The young company just started post-training and will report early results soon; she expects plenty of flaws once models ship.

A refreshingly frank take on benchmark inflation.

Original post →

Startup exec admits pass@k coding benchmarks can be gamed with synthetic data

More from Models