EMNLP paper: retrieval benchmarks like FreshStack stay reliable under temporal drift
mrdrozdov · x · 2026-10-26
An EMNLP 2026 Findings paper (Kuissi, Subrahmanyan, Thakur, Lin) stress-tests retrieval models on FreshStack (LangChain) across two snapshots: Oct 2024 vs Oct 2025. Key findings:
- Drift mostly relocates evidence rather than invalidating it: only 1 of the 2024 queries lost full support in the 2025 corpus, with relevant documents migrating to competitor repos like LlamaIndex
- Model rankings barely shift between snapshots, with Kendall τ correlation up to 0.978 at Recall@50
Conclusion: technical-documentation retrieval benchmarks can remain reliable when re-judged under this drift regime. Code and data are open-sourced as driftbench.
More from Research
- Debate: could exponential compute replace wet-lab experiments in biology? — cjmaddison · 2026-10-26
- LLMs are obsessed with Japan: 24-language study traces cultural bias to SFT — alex_verem · 2026-10-26
- Top-b sampling: entropy-adaptive decoding beats Top-p on GPQA and cuts variance 10x — cneuralnetwork · 2026-10-26
- MathDB launches as an open platform for AI-era math research with Lean support — lpachter · 2026-10-26
- ConformWeb: LLM-generated web apps pass only 11.7% of behavioral contract checks — algo_diver · 2026-10-26
- Claude runs 53 hours unsupervised, beats human designs on 7 of 8 quantum problems — FinanceYF5 · 2026-10-26