Search consultant ditches LLM-as-a-judge for agent-driven synthetic traffic evals

IgorCarron · x · 2026-11-17

Search consultant softwaredoug explains moving away from LLM-as-a-judge evaluation. The old regime — LLMs labeling results, then NDCG-style metrics — bolts AI onto legacy workflows and ignores the search UI's foundational role, so ranking wins often fail to translate into A/B test success. His new approach: vibecode a facsimile of the client's application, let agents act as users solving search use-cases in the UI, and evaluate sessions and clicks instead of NDCG. Framed as a better way to leverage LLM feedback, not a replacement for human testing.

Original post →

More from coding & agent

coding & agent channel →