Average RAG scores can hide regressions: review retrieval changes query-by-query
SeaAstronomer9044 · reddit · 2026-10-17
When comparing two RAG retrieval configurations, looking at the average first can hide a regression on a critical query. The author recommends comparing the same individual queries across runs before averages.
A useful review sorts queries into four groups:
- Queries that improved
- Queries that regressed
- Queries that failed in both runs
- Queries that can't be assessed due to missing or incomplete reference judgments
Evidence for each regression should be specific: which supporting document was expected, where it ranked before and after, and whether it still reached the context window. This helps a team decide what to investigate before a migration — but it doesn't prove which code change caused the regression or that the rest of production is safe.
The author's preferred deliverable: a short decision summary backed by reproducible query-level results, with uncertainties visible.
More from coding & agent
- Backend migration from Python to Go now takes one engineer just one week — rakyll · 2026-10-17
- Audio founder says Claude one-shot decompiled rival algorithms, moat gone — ChrisGPT · 2026-10-17
- Google's playbook for agent evals: four layers of checks beyond the vibe check — rseroter · 2026-10-17
- AgentForesight: a 7B online auditor that catches multi-agent failures mid-run — jiqizhixin · 2026-10-17
- Dev gives his agent an alarm tool to wake him during overnight /goal runs — zack_overflow · 2026-10-17
- Dev shares multi-agent orchestration recipe: 2-3 lead agents with many focused subagents — MoonL88537 · 2026-10-17