Project APE builds its verifier benchmark from 100 AI-written papers with injected errors
soumitrashukla9 · x · 2026-07-22
This post explains a core benchmark-design issue for Project APE: what counts as ground truth?
- The team starts from 100 fully AI-written papers.
- They then inject errors intentionally according to the CRED taxonomy.
- The benchmark asks whether a perfect verifier can detect all of those injected defects.
It is essentially a methodology post about how to evaluate verification systems under controlled, known errors.
Related event: Project APE Evaluates LLMs as Autonomous Research Error Verifiers(5 posts)→
More from Research
- Penn's Kording Lab opens nominations for Spring 2027 speaker series — KordingLab · 2026-10-27
- neuron-explorer adds MLX backend for per-neuron steering on Apple Silicon — arthurcolle · 2026-10-27
- OpenAI says its models produced a Yang-Mills existence proof, formalized in Lean — exophades · 2026-10-27
- Bubeck hails subexponential-time 3SAT algorithm as biggest TCS surprise in 20 years — danintheory · 2026-10-27
- Greg Brockman announces AI-assisted discoveries in math, CS, and quantum field theory — gdb · 2026-10-27
- AI is simultaneously flooding journals with slop and cleaning up human slop — _akpiper · 2026-10-27