Cisco releases Signals 2.0 to auto-diagnose 45 types of AI agent failures, hitting 91.43% F1

ptkbhv · x · 2026-11-11

Cisco launched Signals 2.0, a system that continuously detects AI agent failures in production traces, with an accompanying paper on arXiv.

Approach: Agents can make wrong decisions or waste resources even when tools work fine; nondeterministic behavior makes these hard to reproduce. Signals mines agentic traces to surface issues and suggest concrete fixes.
Method: Built on a taxonomy of 45 problem types across 8 families, each with definitions and evidence requirements; distinguishes realized failures, risk exposures, and inefficiencies, groups findings into engineering issues called Signals, and assesses severity, pairing each with actionable remediations and verification checks.
Results: Achieved 91.43% F1 matching Signals to reference issues on public agent executions—about 18 points higher than a one-pass LLM control using the same model and taxonomy.

More from coding & agent