incident-response

When to Use

When a CloudWatch Alarm or Prometheus AlertManager sends a threshold-exceeded alarm
autopilot-deployImmediately after the circuit breaker trips and a rollback is executed
continuous-eval When a regression gate failure occurs twice in a row
When the user explicitly /incident-response <alarm-id>to request root cause analysis

Exclusions:

Transient errors recoverable by a simple retry (circuit breaker self-recovery)
Legacy environments without diagnostic data (fall back to manual runbooks when the MCP data layer is absent)

Prerequisites

CloudWatch Alarms + Prometheus AlertManager alarm source configuration.
awslabs.cloudwatch-mcp-server==0.0.25, awslabs.prometheus-mcp-server==0.2.15 MCP access (@latest prohibited, PyPI version pin required).
Runbook repository: .omao/plans/runbooks/ 에 ${symptom}.md stored in the format.
On-call routing integration such as PagerDuty / Opsgenie / Slack (for automatic SEV1 paging).
autopilot-deployaccess permission to the state file (.omao/state/autopilot-deploy/) — used for deployment freeze.

Severity Classification Criteria

Received alarms are immediately classified according to the following criteria.

Severity	Criteria	Human intervention
SEV1	Positive Toxicity/PII leakage, data exfiltration, full production outage, 30% or more traffic errors	Immediate on-call page. The Agent performs diagnosis only and does not execute remediation.
SEV2	Partial service outage, P99 latency increase of 2× or more, circuit breaker trip, specific region outage	Agent prepares a drafted response → executed after human approval.
SEV3	Quality regression (faithfulness -5pp, etc.), cost spike, increased error rate for a single agent	Agent prepares a drafted response → executed after human approval.
SEV4	Warning-level (increased log volume, 15% increase in token usage, etc.)	Agent only generates a report and adds it to the weekly review queue.

5-Step Response Playbook

Step 1: Receive & Classify

Upon receiving an alarm, parse the payload and finalize the severity.

ALARM_ID="$1"
ALARM=$(aws cloudwatch describe-alarms --alarm-names "$ALARM_ID" --query 'MetricAlarms[0]' --output json)

# Or via MCP
# mcp__cloudwatch__get_alarm --name "$ALARM_ID"

SEVERITY=$(jq -r '.Tags[] | select(.Key=="severity") | .Value' <<< "$ALARM")

If the severity is not finalized, treat it as the default SEV3 and request human confirmation.

Step 2: Runbook Lookup

Based on symptom keywords, .omao/plans/runbooks/ search for a response runbook in

SYMPTOM=$(jq -r '.AlarmDescription' <<< "$ALARM" | sed 's/[^a-z0-9-]/-/g')
RUNBOOK=$(ls .omao/plans/runbooks/*.md | grep -i "$SYMPTOM" | head -1)

if [ -z "$RUNBOOK" ]; then
  echo "No matching runbook. Proceeding with generic diagnostic flow."
fi

If a runbook exists, follow its steps; if not, proceed to the Step 3 generic diagnostic flow.

Step 3: Hypothesis Generation

Generate 3–5 hypotheses based on the runbook's "Possible Causes" or generic rules. Each hypothesis must be mapped as a pair with a diagnosable MCP query.

{
  "hypotheses": [
    {
      "id": "H1",
      "claim": "Retrieval index outdated after 2026-04-20 reindex job",
      "diagnostic_query": "cloudwatch: /aws/lambda/reindex-job last 24h",
      "confidence_prior": 0.4
    },
    {
      "id": "H2",
      "claim": "New model version v2.3.1 introduced context window truncation",
      "diagnostic_query": "prometheus: agent_context_truncation_total{version='v2.3.1'}",
      "confidence_prior": 0.3
    },
    {
      "id": "H3",
      "claim": "Vector DB (Milvus) slow query due to compaction backlog",
      "diagnostic_query": "prometheus: milvus_compaction_queue_length",
      "confidence_prior": 0.3
    }
  ]
}

Step 4: Diagnostic MCP Queries

Execute the MCP queries corresponding to each hypothesis in parallel.

# Hypothesis H1: reindex job health
mcp__cloudwatch__filter_log_events \
  --log-group /aws/lambda/reindex-job \
  --start-time $(date -u -d '-24 hours' +%s)000 \
  --filter-pattern "ERROR"

# Hypothesis H2: context truncation
mcp__prometheus__query_range \
  --query 'rate(agent_context_truncation_total{version="v2.3.1"}[5m])' \
  --start "$(date -u -d '-6 hours' +%s)" \
  --end "$(date -u +%s)"

# Hypothesis H3: Milvus compaction
mcp__prometheus__query \
  --query 'milvus_compaction_queue_length'

Based on the results, update the posterior confidence of each hypothesis. Finalize the highest-confidence hypothesis as the root cause candidate.

Step 5: Remediation — Drafted Response + Human Approval

For SEV2/3, create the remediation command as a drafted file in .omao/state/incident/${id}/remediation.sh . It is not executed automatically.

cat > .omao/state/incident/sev2-20260421-1023/remediation.sh <<'EOF'
#!/bin/bash
# Proposed remediation for SEV2 incident
# Root cause: Milvus compaction backlog (H3 confidence 0.82)
# Reviewer: please approve before execution

kubectl -n milvus exec milvus-proxy-0 -- milvus-cli \
  --command "compact -collection=agent_kb"

# Verify
kubectl -n milvus exec milvus-proxy-0 -- milvus-cli \
  --command "describe -collection=agent_kb" | grep "compaction_state"
EOF

echo "Drafted remediation at .omao/state/incident/sev2-20260421-1023/remediation.sh"
echo "Approve via: gh issue comment <issue-id> --body '/approve-remediation'"

For SEV1:

# Page on-call immediately
curl -X POST "$PAGERDUTY_INCIDENT_URL" \
  -H "Authorization: Token token=$PD_TOKEN" \
  -d "$(jq -n --arg id "$ALARM_ID" --arg desc "$SEVERITY $SYMPTOM" \
    '{incident:{type:"incident",title:$desc,service:{id:"PXXXXX",type:"service_reference"},urgency:"high"}}')"

# Freeze autopilot-deploy
echo '{"circuit_breaker_status":"tripped","reason":"SEV1 incident"}' \
  > .omao/state/autopilot-deploy/freeze.json

State Management

All incidents are recorded in the .omao/state/incident/${severity}-${timestamp}/ directory.

timeline.jsonl — Timestamp of each step start/completion
hypotheses.json — Generated hypotheses and posterior confidence
diagnostic-results/ — Raw MCP query results
remediation.sh — Drafted recovery commands (SEV2/3)
postmortem-draft.md — Post-mortem draft automatically generated after incident closure

Example Inputs/Outputs

Input: /incident-response rag-qa-error-rate-spike

Output (SEV2):

[12:35Z] Received alarm: rag-qa-error-rate-spike
[12:35Z] Severity: SEV2 (error rate 2.1× baseline for 5m)
[12:35Z] Runbook match: .omao/plans/runbooks/rag-qa-error-spike.md
[12:36Z] Generated 3 hypotheses
[12:38Z] Diagnostic MCP queries complete
[12:38Z] Root cause candidate: H3 (Milvus compaction backlog, confidence=0.82)
[12:39Z] Drafted remediation at .omao/state/incident/sev2-20260421-1235/remediation.sh
[12:39Z] AWAITING HUMAN APPROVAL. autopilot-deploy frozen.

Output (SEV1):

[14:02Z] Received alarm: pii-leak-detected
[14:02Z] Severity: SEV1 (PII token found in agent response)
[14:02Z] On-call paged: PagerDuty incident P-A1B2C3
[14:02Z] autopilot-deploy frozen for all agents
[14:02Z] Agent diagnosis continuing; no remediation will be auto-drafted for SEV1
[14:05Z] Diagnostic complete: see .omao/state/incident/sev1-20260421-1402/
[14:05Z] Human responder in control. Agent awaiting /release-sev1 command.

References

Official documentation

CloudWatch Alarms — Alarm definition
Prometheus AlertManager — Alarm routing
awslabs/mcp — cloudwatch-mcp-server — CloudWatch MCP
PagerDuty Events API v2 — On-call routing

Engineering blog

Google SRE Book — Managing Incidents — Incident command principles
Incident.io — Hypothesis-driven debugging — Hypothesis-driven diagnosis patterns

About this skill