Evaluating Agents Without Right Answers: Similarweb's Playbook
LangChain · x · 2026-07-30
Similarweb detailed their engineering experience using LangSmith to evaluate long-form Agent research reports.
The core challenge is that unlike traditional software with clear pass/fail tests, agentic systems can take different paths and produce varying valid outputs for the same input. To tackle this, they adopted a multi-dimensional evaluation strategy:
- Deterministic checks: Verifying if tool calls are correct.
- Rubric-scored LLM judges: Scoring generation quality using rubrics.
- Faithfulness checks: Ensuring content is faithful to retrieved data sources.
- Baseline A/B comparisons: Comparing new versions against a saved baseline.
The author emphasizes treating scores as signals, not absolute answers, and calibrating rubric weights carefully before use to avoid misjudging system improvements as regressions.
More from coding & agent
- llamppl: Integrating Large Language Models into Probabilistic Programming — xuanalogue · 2026-07-30
- DeepLearning.AI Launches Free AI Code Review Course to Build Custom Review Agents — iamKierraD · 2026-07-30
- Perplexity Open-Sources Numbat: A Cross-Framework Agent Detection and Response Layer — AravSrinivas · 2026-07-30
- Google Cloud Tutorial: Build and Deploy Agents via Agents CLI in Your Coding Harness — rseroter · 2026-07-30
- Invideo Launches Agent Two for AI Filmmaking Workflows — gen_ericai · 2026-07-30
- Cerebras on the Agentic Era: New Workflows Will Drive Non-GPU Chip Architectures — sarahookr · 2026-07-30