incident-triage

Incident Triage

Decision framework for the first 5 minutes: classify → scope → route. No deep diagnosis.

Relationship with other skills

• incident-triage is the entry point — after a quick assessment, route to the specific skill
• SEV1/2 diagnosis → aws-troubleshooting(deep investigation)
• Need to check logs → log-diagnosis
• Need a full inspection → health-check

Workflow

Step 1 — Collect signals (< 1 minute)

MUST:

• If triggered by an alert:
  • Read the alert details:aws cloudwatch describe-alarms --alarm-names {name} --region {region}
  • Record metric, threshold, duration, current value
• If reported by a user:
  • Confirm the symptoms (what is not working)
  • Confirm the start time (when it started)
  • Confirm the scope of impact (who is affected)
• Quickly check the overall current alert landscape:
  • aws cloudwatch describe-alarms --state-value ALARM --region {region}
  • Check all the account's main regions
• Check change events in the last 15 minutes:
  • aws cloudtrail lookup-events --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false --start-time {T-15m} --region {region}

SHOULD:

• Check AWS Health Dashboard:aws health describe-events --filter '{"eventStatusCodes":["open","upcoming"]}' --region us-east-1
• If there is a dashboard, quickly look at the trends of key metrics

Step 2 — Determine Severity

MUST:

• Read references/severity-matrix.md to classify
• Judge based on evidence, not speculation

Classification quick reference:

• SEV1: service completely unavailable / data risk → act immediately
• SEV2: service degraded / partially affected → act within 15 minutes
• SEV3: abnormal observation / service not affected → act within 1 hour

Step 3 — Scope the impact

MUST:

• Determine the blast radius: single resource → single service → single AZ → single region → cross-region
• Check related services (refer to the dependency chain list in aws-troubleshooting/references/investigation-framework.md)

SHOULD:

• Use health-check for a quick global inspection to confirm whether there is a cascading impact
• Check whether it is an AWS-side event (Health Dashboard)

Step 4 — Route + output

MUST:

• Output an incident report (assets/incident-report-template.md format)
• Include: severity, blast radius, signal summary, preliminary assessment, recommended actions

Routing rules:

• SEV1 → route to aws-troubleshooting (query aws-knowledge MCP + SSM Automation runbooks in real time)
• SEV2 + need logs → route to log-diagnosis
• SEV2 + need metrics → route to health-check
• SEV3 → route to health-check for a full inspection

SHOULD:

• Provide specific mitigation commands for SEV1/2 (but mark them human-required, do not execute automatically)

Query reference

• Severity classification rules:references/severity-matrix.md
• Cross-service dependency chain:../aws-troubleshooting/references/investigation-framework.md
• Service-specific troubleshooting SOP: aws-troubleshooting skill (query aws-knowledge MCP in real time)

Automation Boundary

auto-safe

• aws cloudwatch describe-alarms(check alert status)
• aws cloudtrail lookup-events(check recent changes)
• aws health describe-events(check AWS-side events)
• describe / list / get type resource queries
• Generate an incident report

human-required

• All mitigation/repair operations (restart pods, scale out, roll back, modify configuration)
• Open an AWS Support Case
• Notify others

Gotchas

• CloudTrail events have a ~5 minute delay; the most recent changes may not have appeared yet
• aws health API only has global events in us-east-1; other regions only have region-specific events
• An INSUFFICIENT_DATA alert is not necessarily a problem—it may be a newly created alert that has not yet received enough data
• Brief alerts during deployment (rolling update) are normal; check whether it is within the deployment window
• Do not spend more than 5 minutes analyzing a SEV1—if the root cause cannot be found within 5 minutes, escalate