Afraid to touch your RAG prompt? A 5-step eval loop from a real FAQ bot
Al_Grigor · x · 2026-10-21
Alexey Grigorev (DataTalksClub) shares a lesson from shipping an FAQ bot for their courses: without evals, you're changing blind. After the bot launched, tiny fixes piled up and the system prompt got longer and noisier — but he was afraid to clean it up in case something broke.
So he added evals, and Paul Iusztin turned it into a 5-step case study:
- Collect the data, even if you never traced it
- Build the eval set (61 cases in the end)
- Build the evaluators
- Run them in an eval harness
- Close the loop
The core message: move from vibe checks to a score you can actually ship on.
More from coding & agent
- Voice agent developers wrestle with the hardest edge case: knowing when to hang up — Kindly-Duty272 · 2026-10-21
- OpenCode hacked, eMudra used private key of 4, PHP hit by supply-chain chaos — burny_tech · 2026-10-21
- Falconer launches to unify internal knowledge bases and docs for agents, raises from GC and BoxGroup — heyshrutimishra · 2026-10-21
- PHP ships emergency fixes for two 9.2/10 RCE vulnerabilities — MickeySteamboat · 2026-10-21
- Anthropic releases Claude Sonnet 5.5: Terminal-Bench jumps from 10.3% to 70.6%, tasks up to 30% cheaper — Aiden_Tech_Ai · 2026-10-21
- Indie hacker ships 8 email product features in one drop with Opus, shares inbox open-rate data — marclou · 2026-10-21