RAG evals can fail even when your system is correct — leave-one-out fix
Al_Grigor · x · 2026-10-31
Alexey Grigorev shares a counterintuitive RAG eval pitfall: his harness replays historical student questions, and a question correctly classified as NEW months ago is now correctly flagged DUPLICATE once it's in the knowledge base — yet the eval reports failure. His fix: leave-one-out evaluation, removing the FAQ entry created by the original question before replaying it, with experiments tracked in Opik. Lesson: reproduce the original data state, not just the original prompt.
More from coding & agent
- One malformed item killed my agent pipeline — now loading repairs it instead — petrusenko_max · 2026-10-31
- Letting an agent pay for and renew its own API access with x402 and a budgeted wallet — Minimum_Hour519 · 2026-10-31
- Open-source Orkas orchestrates 16 desktop AI agents, gains 1,000 stars in a day — Orkas-AI · 2026-10-31
- Free Claude Code skill auto-edits videos and records website B-roll for you — TawohAwa · 2026-10-31
- Row-Bot 5.3.1 adds linked conversations, no-cost waits, and timeline video editing — Acceptable-Object390 · 2026-10-31
- Agent guard test flaw: a guard you can't test off its host is just a launchd check — petrusenko_max · 2026-10-31