LangSmith AI Search: query agent traces by behavior, median Open SWE trace runs 84K tokens
LangChain · x · 2026-10-17
LangChain launched AI Search in LangSmith, letting teams find agent conversations by semantic behavior descriptions like "did the user get frustrated?" or "did PII leak?"
- Traces are read within a fixed token budget. On 150 conversations from Open SWE, the team's daily coding agent, the median trace ran 84K tokens — only 1.5K of that was user messages plus the final reply, while roughly 3/4 was tool output (file reads, shell commands, test logs).
- Retrieval pipeline: structured filters (e.g. time range) narrow candidates first, then a small fast model checks each remaining trace against the description in parallel, usually under half a second each, streaming matches as they're found.
- It builds on SmithDB, the trace-purpose-built database, and Custom Apps; trace mining is commonly used to build eval datasets and annotation queues.
Related event: LangChain Launches LangSmith AI Search for Semantic Agent Trace Retrieval(4 posts)→
More from coding & agent
- Sentry CEO: agents unlock far more once you stop assuming prompts are the answer — zeeg · 2026-10-17
- Stop Assuming Prompts Are the Answer — Agents Get Far More Powerful — zeeg · 2026-10-17
- tszzl: everything effectively has a public API — a discoverable functional signature — tszzl · 2026-10-17
- HQ Dev Day: Bring Your Own Bot — External Agents Plug In and Talk to Each Other — jacob_posel · 2026-10-17
- Grok Bot computer use now 3x faster on average, replies arrive in under half the time — Baconbrix · 2026-10-17
- Reflection AI Launches Mirror, an Open Coding Agent Optimized for Beam — sloppenheimer · 2026-10-17