Production Incident Investigation: A Six-Step Playbook from Triage to Postmortem

_jaydeepkarale · x · 2026-11-02

Engineer jaydeepkarale wraps up his Production Debugging Playbook series with Part 6, a structured process for investigating live incidents—arguing that process, calm heads, and blameless culture matter more than raw technical skill.

The six steps:

Triage: what's broken, blast radius, affected journeys, severity based on business impact.
Gather evidence: correlate metrics (Prometheus), logs (Loki/Splunk), traces (OpenTelemetry), K8s events, dependencies, and recent changes (deploys, config, feature flags).
Timeline & scope: reconstruct deploy → latency spike → alerts → incident declaration.
Root cause: isolate across code/release, data layer (slow queries, connection exhaustion), dependencies (DNS, external APIs), and infrastructure (resource saturation, OOM).
Mitigate & resolve: stop the bleeding first (rollback, feature flag, fallback), then fix the root cause, validate in production, and only close after sustained stability.
Learn & prevent: incident report, add tests/alerts, guardrails like canary deploys and SLOs.

Includes a full Detect → Triage → Investigate → Mitigate → Resolve → Learn workflow infographic. The quoted Part 5 covers debugging network/dependency failures where CPU and memory look healthy but p95 latency and 5xx climb.

Original post →

Production Incident Investigation: A Six-Step Playbook from Triage to Postmortem

More from coding & agent