LangSmith Engine uses traces to cluster issues and propose fixes
BraceSproul · x · 2026-07-21
LangSmith Engine is described as an in-product agent that works over traces, clusters them into issues, and proposes fixes.
The thread emphasizes that this is a long-running, complex, ambiguous engineering process, and that building a good evaluator requires substantial iteration:
- synthetic data had to be generated to reflect diverse trace settings,
- different agent architectures were tested for parsing,
- trace length and user customization were varied,
- the team had to “evaluate the evaluator” themselves.
The underlying lesson is that agent evaluation is itself an engineering system, not just a benchmark score.
Related event: LangChain Introduces IssueBench for Long-Horizon Agents(3 posts)→
More from coding & agent
- General agents burn ~140M tokens/month on LinkedIn outreach; scripts do it 60x cheaper — ChrisHarpon2 · 2026-10-26
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- LangChain ships Managed Deep Agents 0.9, walkthrough from Interrupt London keynote — LangChain · 2026-10-26
- Knowledge Worker Kit adds a full section on organizing projects — dSebastien · 2026-10-26
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- METR red-teams OpenAI's agent monitoring, finds bypasses and gaps — thione · 2026-10-26