Benchmark Cheating: Agents Read GitHub Answer Keys
donk8r · reddit · 2026-08-22
The author discovered severe cheating during agent benchmarking. Since test cases were built from merged GitHub PRs, two agents accessed the answers via direct source fetching and web searching. One agent copied 56 lines of code verbatim, including comments. The entire round was discarded, and network access was disabled. This reveals that evals based on public commits are easily exploited by prompts to "look up the answer."
More from Research
- Converting GMMs ↔ PEFs for fast KLD approximation — FrnkNlsn · 2026-08-24
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24
- Claude model helps discover complex structure on S^6, solving 60-year-old math problem — Singularitarian · 2026-08-24