Prime Intellect Audits Public AI Benchmarks, Finds Contamination and Web Access Issues
xeophon · x · 2026-11-04
Prime Intellect audited several popular public evaluation benchmarks and found issues — including problems in AutomationBench and OfficeQA Pro v2, plus flaws around web access in public benchmarks ("we looked at the data and the data stared right back at us"). The team says it has fixed the issues it found and is sharing its findings. The audit took months, more output is promised, and the authors note that while public evals are foundational for assessing model capabilities, monitoring them properly is hard.
Related event: Prime Intellect Audit Exposes Flaws in Popular AI Benchmarks(2 posts)→
More from Research
- NRGPT paper reimagines GPT inference as gradient descent on an explicit energy landscape — burkov · 2026-11-04
- Fireworks event slides show sliding-window bit-sharing quantization trick — sophiamyang · 2026-11-04
- UCLA Study Seeks Users of 'Griefbots' to Interview on AI and Mourning ($30 Gift Card) — Interesting-Cat-7838 · 2026-11-04
- Tiny LoRA adapter beats DNA model Carbon-X on RareBench, exposing generalization limits — danielmckinn0n · 2026-11-04
- QuickProof: formal verification for C with specs written as plain C tests — thomasahle · 2026-11-04
- RSNA 2026 Deep Learning Lab to teach medical imaging VLM dataset building from scratch — PRouzrokh · 2026-11-04