Lyft's Real-World User Evaluation Loop
AI Engineer · youtube · 2026-07-19
Nick Ung discusses how Lyft builds closer-to-production evaluations for its customer service and agent systems.
The core issue is that many offline evals are too "easy"—the "customers" in tests are just standard LLMs that completely lack the frustration, tangents, and adversarial nature of real users, leading to unanticipated failure modes post-launch.
Lyft's approach involves:
- Fine-tuning an adversarial user simulator using real Lyft passenger/driver chats, specifically designed to act emotional, chaotic, and provocative.
- Building a lightweight eval harness that allows engineers to write benchmarks with minimal code.
- Calibrating LLM judge rubrics against human annotations to align labeling consistency.
- Continuously feeding production failure trajectories back into offline test sets, creating an iterative loop.
Their customer service agent handles roughly a third of Lyft's support queries, scaling to millions of conversations per month. This case study highlights how to build evaluation as a sustainable engineering system, rather than focusing on a single model.
More from coding & agent
- Practical tip: use max effort for planning, xhigh for implementation in Claude Code — majidmanzarpour · 2026-10-27
- Chrome team explains WebMCP: agents get tools from your website, not a replacement for MCP — rseroter · 2026-10-27
- Agent-Native apps: why agents bypassing your UI can break business rules — rseroter · 2026-10-27
- 'My intelligence stack': developer splits AI coding work across Opus 5.5, Fable and Astra with human review gates — sh_reya · 2026-10-27
- Baseten webinar: evaluate open models for coding agents by benchmarking on your own merged PRs — baseten · 2026-10-27
- Thorsten Ball on working with agents: run more evals, automate everything, use the best models — omarsar0 · 2026-10-27