Deep research and web retrieval benchmarks are stale — Reddit calls for real-environment evals
Lucky_Creme_5208 · reddit · 2026-09-26
- A Reddit user argues most benchmarks are either saturated or tested under artificially restricted conditions — e.g., AA Omniscience restricts tool access.
- Deep research, wide web search and fact retrieval are among the most common chatbot use cases, yet dedicated benchmarks are scarce, and the few that exist have no official leaderboard and haven't been updated in years.
- The post calls for a benchmark run in real environments — with tool access, internet search, and real harnesses like Claude or ChatGPT Work — and asks the community whether one exists.
Related event: Community Calls for Real-World Search Benchmarks as Existing Ones Saturate(2 posts)→
More from Research
- Reprogramming fewer than 10 dormant neurons backdoors any LLM agent, no training needed — DimitrisPapail · 2026-11-11
- Scandinavian Journal of Statistics editors reflect on publishing in the age of generative AI — michaelchchoi · 2026-11-11
- Proving LLM backdoor-safety without revealing weights: ZKML over formal methods — DimitrisPapail · 2026-11-11
- LLMs can backdoor other LLMs by reprogramming a few dormant neurons to trigger rm -rf — DimitrisPapail · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11