EviCast Forecasts RAG Retrieval Gains Before Deployment, Cuts Diagnostic Calls by 80%
_reachsumit · x · 2026-11-04
A new arXiv paper presents EviCast, a framework that forecasts whether retrieval will help an LLM before large-scale deployment.
- Key idea: via interventional evidence-state decomposition, it profiles the generator's response to controlled evidence states once, then combines this shared profile with each candidate retriever's offline evidence-state distribution to estimate end-to-end net utility and its 95% lower confidence bound — without executing every RAG configuration.
- Motivation: relevant evidence doesn't guarantee the generator can use it; indiscriminate retrieval can hurt performance while adding retrieval and long-context inference costs.
- Results: correctly decides all 14 HotpotQA retrieval configurations with 2.75 F1 forecast error while cutting diagnostic generation calls by 80%, the lowest error among seven predictors.
A state-refinement analysis characterizes when its binary and topology-aware instantiations are exact and when shared calibration beats per-retriever calibration statistically.
More from Research
- Noetra engineer's first project: building an image encoder from scratch for multimodal LLMs — HirokatuKataoka · 2026-11-04
- Japanese team trains image encoder from scratch to match SigLIP2, with better CJK performance — HirokatuKataoka · 2026-11-04
- SpecJudge author argues LLM-as-judge tools should never ask models for self-reported confidence — jokiruiz · 2026-11-04
- Tsinghua's BEMULA benchmark: no LLM agent memory system passes over 1.5% of forget requests — rohanpaul_ai · 2026-11-04
- KAIST AI's FactEmbed decouples query understanding to improve composed image retrieval — kaist-ai · 2026-11-04
- Intology's Locus auto-designs molecular models, beating GPT-6 and Claude agents — rohanpaul_ai · 2026-11-04