Project APE builds its verifier benchmark from 100 AI-written papers with injected errors
soumitrashukla9 · x · 2026-07-22
This post explains a core benchmark-design issue for Project APE: what counts as ground truth?
- The team starts from 100 fully AI-written papers.
- They then inject errors intentionally according to the CRED taxonomy.
- The benchmark asks whether a perfect verifier can detect all of those injected defects.
It is essentially a methodology post about how to evaluate verification systems under controlled, known errors.
Related event: Project APE Evaluates LLMs as Autonomous Research Error Verifiers(5 posts)→
More from Research
- Interpretability researcher mathildepapillo joins Goodfire post-PhD, calls interp the key problem — mathildepapillo · 2026-10-27
- Reasoning models can barely control their chain of thought, paper finds — tomekkorbak · 2026-10-27
- New paper: LLM data schedules and learning rate schedules should be co-designed — jasondeanlee · 2026-10-27
- What makes research get adopted: important problems, proper evals, and usability — srchvrs · 2026-10-27
- Bitter lesson part 2: few academic inventions matter, and conference publications aren't the key — srchvrs · 2026-10-27
- First-week PhD student lands ARR Area Chair role, exposing peer-review quirks — anmarasovic · 2026-10-27