Eval-driven development for RAG: keep eval cases in Git, never regress
Al_Grigor · x · 2026-10-15
Engineer @AlGrigor shares how he evaluates an AI FAQ curation pipeline for DataTalksClub with a deliberately simple harness: eval cases live in Git as the source of truth (61 versioned cases), runs are traced in Comet's Opik tracking classification accuracy, placement accuracy and cost, and every failure becomes a new eval case. The goal isn't 100% pass rate — it's making sure today's fix doesn't break yesterday's working behavior, shipping only when metrics don't regress. Full eval-driven-development flowchart included.
More from coding & agent
- Swamp Club raises $7M seed for behavior-centric AI agent platform — dauber · 2026-10-15
- Coinbase for Agents goes live on Composio, letting AI agents trade crypto, stocks and derivatives — kleffew94 · 2026-10-15
- Demo shows Gemini completing a 6-year full-stack upgrade and cloud deploy in one go — rseroter · 2026-10-15
- TV noise drops voice agent task completion from 38.7% to 8.6%; audio cleanup model helps — deliprao · 2026-10-15
- Running 40+ agents locally turns into an I/O and compute management problem — rachittshah · 2026-10-15
- AI Agent Swarm Dynamics Hackathon names winners; top team found unknown Chinese agent swarm — lfschiavo · 2026-10-15