From vibe checks to a 61-case suite: a practical RAG evals workflow
dl_weekly · x · 2026-10-28
Alexey Grigorev (DataTalks.Club founder, 15-year engineer) shares his end-to-end RAG evals workflow for a FAQ chatbot serving 100k+ learners. Key insight: "without evals you are blind" — yet most teams never record traces from day zero, so building a gold dataset stays a dream. The 5-step path: collect proxy data when no traces exist, turn past failures into eval cases, build evaluators with code checks + LLM judges, run a 61-case suite on every prompt/model change, and close the loop by tracking cost and quality in Opik. Based on an interview with real production lessons.
More from coding & agent
- Ziplime: new Python library for AI-driven algorithmic trading and backtesting — mdancho84 · 2026-10-29
- Code ownership in the AI era: you land it, you own it, bot approvals never count as human ones — ccerrato147 · 2026-10-29
- How Anthropic Reviews Code When Claude Writes 90% of It — ccerrato147 · 2026-10-29
- Leaked prompt makes Claude Code build and maintain its own skills automatically — Aiden_Tech_Ai · 2026-10-29
- Top reasoning effort costs up to 42x more on tasks low effort already solves — zainhas · 2026-10-29
- Vercel's open-source Zig coding agent fx ships v0.0.15 with 70% less memory, 6.15 MiB binary — evilrabbit_ · 2026-10-29