Improving Agents is a Data Mining Problem, Says LangChain Engineer
AI Engineer · youtube · 2026-08-13
Vivek Trivedy from LangChain argues that observability and continual learning are fundamentally the same problem. When an agent acts in an environment, the traces it produces are the only real record of what happened and the substrate for everything else.
Key Insights & Practices
- Mining Traces: Agents often get dumber after context compaction. You cannot read this off the code; you have to analyze the traces. LangChain points agents at the traces of other agents to find where users got upset or how a different model would have behaved.
- Cost & Model Choice: Working with Harvey on a legal benchmark, they found an open model could match a frontier model's trace judging at 1-2 orders of magnitude lower cost through harness engineering.
- When to Finetune: His rule is based on feedback speed: exhaust harness engineering (which gives feedback in 2 minutes) first, finetune to break through the ceiling, then return to harness engineering.
- Dense Feedback: What agents lack most is dense feedback. A simple pass/fail benchmark gives an agent nothing to act on, whereas traces hold fine-grained signal.
- Evals Describe Behavior: You can describe an agent's behavior simply by showing the evals it was measured against, because those are what it hill climbs.
More from coding & agent
- Build a 3D Book Render Tool with a Single Claude Prompt — burkov · 2026-08-13
- New site shows real examples from popular coding benchmarks to make scores concrete — mishig25 · 2026-08-13
- W&B Weave Introduces BYOB: Track Multimedia via S3/GCS URIs Directly — wandb · 2026-08-13
- Smuggling Prompt Injections Using Zero-Width Characters to Bypass Filters — GiiTZzz · 2026-08-13
- Parahelp Founders' Journey: Pivoting to AI Support, Resolving Millions of Tickets for Cursor — ycombinator · 2026-08-13
- AI Reshapes Design: Prototypes Are Now Production Front-End Code — MxMnr · 2026-08-13