Designing an AI Agent for Kubernetes Incident Response: Noise Reduction, Lower MTTR, Guardrails

_jaydeepkarale · x · 2026-11-05

A developer shared his interview answer for designing an AI agent for Kubernetes incident response, with a full architecture:

Three goals

Reduce noise: correlate alerts and filter known non-actionable ones
Reduce MTTR: let the agent do repeat work (events, logs, metrics, runbooks) and hand over a diagnosis when a human is needed
Auto-resolve low-risk issues; everything else needs human approval

Architecture

Prometheus Alertmanager fires alerts; email and PagerDuty stay untouched—the agent is an addition, not a replacement
SNS → SQS (buffering, retries, DLQ) plus deduplication so one incident doesn't trigger 50 agent runs
Lambda validates source/environment; Step Functions/AgentCore orchestrates the workflow
Agent built on Strands Agents SDK with an Amazon Bedrock knowledge base
MCP Gateway provides read-only access to EKS, Grafana, and CloudWatch MCP servers
Policy engine + guardrails: no deleting deployments/statefulsets/secrets, delete pods only when healthy replicas exist, least-privilege RBAC, rate limits, logs treated as data not instructions
OpenTelemetry tracing and evaluation: incident replay, diagnostic accuracy, pages avoided, MTTR, cost per run

More from coding & agent