Langfuse workshop: turning agent failures into evals with traces, judges and a closed loop
AI Engineer · youtube · 2026-10-12
At the AI Engineer workshop, Annabell Schäfer and Lotte Verheyen of ClickHouse demo a monitoring workflow with Langfuse and a sample IT support agent built on the OpenAI Agents SDK.
Approach
- Capture full traces (observations, model calls, tools) so failures live as signals, not anecdotes
- Three online monitors: an LLM judge for user disagreement, a code-based all-caps detector, and a scope check comparing requests to the system prompt — each with explicit variable mappings
- Seeded production traces let a coding agent investigate patterns via the Langfuse CLI, surfacing unnecessary tool calls, retries, latency and eval coverage gaps
Loop: turn observed failures into representative datasets, run offline experiments, check regressions, repeat. Resources at langfuse.com/workshop.
More from coding & agent
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- Capy 0.5.0 lets Claude Pro/Max subscribers reuse their plan quota, adds OIDC cloud access — garrytan · 2026-10-23
- You.com CEO Richard Socher: legacy code is human slop, let AI clean it up — RichardSocher · 2026-10-23
- Let agents design a single-line glance component so you can track them from the sidebar — blixt · 2026-10-23