One Prompt Line Made an Agent 17 Points Safer and 17 Points Worse
AI Engineer · youtube · 2026-10-25
Sofia Jakovcevic, forward deployed engineer at Arize AI, shows how a one-line prompt change made a support agent 17 points safer but 17 points worse at completing tasks — agents fail at runtime, not in code. She walks through a five-step regression loop built from production traces: curate datasets, score with evaluators, run baseline experiments, compare changes, and gate merges in CI, then demos coding agents that read traces and fix themselves at agent speed.
More from coding & agent
- MCP Sage pipes whole codebases into Gemini 2.5 Pro for second-opinion code reviews — modelcontextprotocol · 2026-10-25
- Microsoft open-sources Skill Recorder: record a task once, your Copilot agent learns to do it — udmrzn · 2026-10-25
- Warp CEO on building AI-native: every task, from tax compliance on, must be agent-runnable — rohanpaul_ai · 2026-10-25
- ra-jit compiles most large Rust projects with incremental builds up to 92x faster than cargo — julianweisser · 2026-10-25
- A prompt trick: ask for "progressive disclosure of complexity" to declutter dashboards — dfeinition · 2026-10-25
- Designer vibe-codes BUNNY'S HOUSE, a smart-home app named after her dog — enggirlfriend · 2026-10-25