Search consultant ditches LLM-as-a-judge for agent-driven synthetic traffic evals
IgorCarron · x · 2026-11-17
Search consultant softwaredoug explains moving away from LLM-as-a-judge evaluation. The old regime — LLMs labeling results, then NDCG-style metrics — bolts AI onto legacy workflows and ignores the search UI's foundational role, so ranking wins often fail to translate into A/B test success. His new approach: vibecode a facsimile of the client's application, let agents act as users solving search use-cases in the UI, and evaluate sessions and clicks instead of NDCG. Framed as a better way to leverage LLM feedback, not a replacement for human testing.
More from coding & agent
- CMU paper: planning agents should show plans, not ask — acceptance jumps from 12% to 55% — rohanpaul_ai · 2026-11-17
- Boring Marketer open-sources 32 marketing agent skills and launches forward-deployed service — boringmarketer · 2026-11-17
- Pybot experiment write-up coming: how the autonomous GitHub bot actually ran — chrisalbon · 2026-11-17
- Running a company in 2031: a meme of one human in a meeting with nine Claude agents — andrew_n_carr · 2026-11-17
- Perplexity to upgrade its Slack agent Computer into a persistent assignable Teammate — AravSrinivas · 2026-11-17
- One chart mapping voice AI: S2S vs cascaded, half- vs full-duplex — rdesh26 · 2026-11-17