Claude finds 36% more benchmark mentions in a rerun, exposing gaps in eval reporting standards
evijit · x · 2026-11-13
Jason Wei notes his initial scan missed many evals: having Claude search harder surfaced 36% more benchmark mentions, though it still misses fancy-UI blog posts and long PDFs — joking that we need a benchmark for benchmark benchmarking. Quoting evijit says their ICML paper EvalEval relied on manual annotation and actually reading all tech reports, arguing the eval reporting landscape needs common-sense community standards like EEE.
More from Models
- METR data on Anthropic employee speedups validates vibe-testing, says Mollick — emollick · 2026-11-13
- Google's Gemini 4 Hydrogen outscores bigger Argon just four weeks after launch — The Decoder · 2026-11-13
- Kaggle launches Arenas: community-voted head-to-head AI model leaderboard — amankhan · 2026-11-13
- Screenshot shows Claude Code account disabled mid-task for 'Terms of Service violation' — cryptodunck · 2026-11-13
- Dev begs AI industry to bring back small/medium/large model naming — andpoul · 2026-11-13
- Same Ghibli 3D prompt: Gemini 4 Hydrogen clearly out-builds Opus 5 — LightAppropriate624 · 2026-11-13