TEMPO benchmark: best of 16 retrievers hits only 32.0 NDCG@10; RAG with no retrieval scores higher
mageed · x · 2026-10-25
Researchers at the University of Innsbruck (with UBC) present TEMPO at EMNLP 2026, a multi-domain benchmark for temporally reasoning-intensive retrieval: real questions often ask "How has X changed since 2017?" rather than "When did X happen?"
- 1,730 human-written queries from Stack Exchange across 13 domains (Bitcoin to history), with 3,976 retrieval steps mapped to gold documents
- Across 16 retrievers, the best (DiveR) reaches only 32.0 NDCG@10; top systems miss part of the required time span on 30% of cross-period queries
- Most striking: in RAG, answering with no retrieval at all scored 77.3, above the best retriever (75.6); gold documents reach 80.5. Off-period evidence likely misleads the generator, and GPT-4o agentic pipelines add little
Oral presentation: Sunday, Oct 25, 11:00–12:30, Room G7-8, EMNLP 2026 Budapest.
More from Research
- New paper: Binary Discrete Diffusion denoises candidate token sets for language reasoning — LucaAmb · 2026-10-25
- Pipeline turns 34 hours of ATP match video into biomechanics data; Claude gives surprisingly good tennis coaching — FinanceYF5 · 2026-10-25
- DeepMind 5-month study: same proactive AI agent was a 'lifesaver' to some, stress to others — rohanpaul_ai · 2026-10-25
- LAW: a language model built purely on self-conditioning takes contrarian diffusion-LM stance — caglarml · 2026-10-25
- Activation steering recipe turns Qwen 3.5 4B into "evil qwen" by nudging two neurons — rachittshah · 2026-10-25
- New dataset reveals humans sharply disagree on when AI agents should speak up in group chats — amyxzh · 2026-10-25