Researcher: perplexity evals favor code-only models; pass@k would be far more informative
eliebakouch · x · 2026-10-24
HF researcher eliebakouch argues it's expected that code-only-trained models beat larger general models on perplexity, making another PPL eval of limited value. He suggests reporting pass@k across coding benchmarks instead, which better reflects real-world coding ability.
More from Research
- 1979 paper dismissed formal verification; 2026 forecast says it'll be as common as fuzzing — danbri · 2026-10-24
- NVIDIA and UW Protein Design team up to release RFD4-Proteina for designed biology — AllThingsApx · 2026-10-24
- T²Quant: task- and timestep-aware mixed precision for quantizing diffusion LMs — Ofek Avivi · 2026-10-24
- Berkeley & AWS: agents return faster code with Lean proofs of equivalence, 1.85x mean speedup — xiuyu_l · 2026-10-24
- Google Discover's SDF filters stale articles before ranking — stale reports down 54.9% in two years — gaganghotra_ · 2026-10-24
- Google says Gemini 4 Argon found fusion counterexamples and designed 5 lab-verified enzymes in hours — Eric520CC · 2026-10-24