Hugging Face ran AI agents on 2,200 papers; some claims didn't hold up
evijit · x · 2026-10-30
Hugging Face researcher @evijit revealed that the Gradio team ran AI-agent replication on 2,200 papers for ICML, and some claims in those papers did not stand up.
The team contacted the authors; some made changes on arXiv, others reran their experiments. She proposes a schema requiring anyone reporting an evaluation result to list settings like temperature, top P and top K, making evaluations machine readable and therefore machine replicable—since many paper claims are no longer verifiable by humans.
More from coding & agent
- Anthropic launches Claude Duet, a proactive agent powered by Opus 5.5 — joellewenstein · 2026-10-30
- The Future of MCP: 6 Shifts That Will Shape Production AI Agents — MaryamMiradi · 2026-10-30
- Spring AI 2.1 adds VectorStore.upsert for apps that bring their own embeddings — therealdanvega · 2026-10-30
- Auditing an LLM-built knowledge graph: 0 of 186 facts matched verbatim, 1/3 of edges hit wrong entities — JeremyCMorgan · 2026-10-30
- Same agent harness, opposite rankings: TB4 benchmark drama exposes eval chaos — lunwang1996 · 2026-10-30
- The eternal eval debugging meme: ask the vendor if it's data or harness, it's data — dejavucoder · 2026-10-30