Get production healthy fast and safely, in the right order: recover first, root-cause second. A live incident is not the time to debug in prod or to fix forward a change you don't understand yet — revert to the last known-good, then investigate calmly.
North star: shortest safe path back to known-good, a clear status thread while it's open, and a written follow-up so it doesn't recur — without a panicked command making it worse.
Step 0 — Triage (60 seconds)
- What's the blast radius? One endpoint, one tenant, or everything? How many users affected?
- What changed most recently? The last deploy is the prime suspect — check deploy history,
git logon the release branch, and recent config/flag changes. - Open a status thread now in your incident channel: one line — what's wrong, who's on it, "investigating." Update it, don't spawn new messages.
Step 1 — Stop the bleeding (recover before you diagnose)
Discover the project's own rollback path — the safe options, cheapest first:
- Roll back the release — redeploy the previous known-good revision. Most platforms (ECS, Cloud Run, Vercel, Fly.io) keep prior revisions one command away.
- Revert the change — if a specific merge caused it, revert that commit and ship the revert, rather than a speculative hotfix.
- Flip the flag off — if the break is behind a feature flag, disable it and redeploy to hydrate. Fastest when it applies.
- Relieve pressure — scale up replicas or bump a stuck queue/concurrency limit if it's load, not a bug.
Do the minimum that restores service. Then confirm recovery (health endpoint, error rate falling, a real request succeeding) and update the status thread: "mitigated."
⚠️ Do NOT make it worse
- Never scale to zero to "restart" — that's 100% downtime.
- Never run a destructive infra apply (destroy, refresh, or apply from an unmerged branch) mid-incident — state churn compounds the outage.
- Never force-delete a secret to "reset" it — the service won't boot without it.
- Don't migrate or roll back the database as a reflex — a schema change mid-incident is high-risk; only if the migration is the cause and the down-migration is proven safe.
- Don't debug in prod by editing live config — reproduce after you've recovered.
Step 2 — Diagnose (now that it's stable)
- Logs + errors — your error tracker for the exception + stack; platform logs around the failure window; alerting context if you have it wired up.
- Correlate to the change — does the error start exactly at the deploy/flag time? What did that change touch?
- Reproduce off-prod (local/staging) to confirm root cause before writing the real fix.
- Common silent causes: an unseeded or misformatted secret, a column too small for new data, an expired token, a config flag that no-op'd on a bad edit.
Step 3 — Fix forward through the normal gates
Once you understand it: write the real fix, run /gate, get it reviewed, and deploy through the normal
pipeline — the fix ships with the same discipline as anything else, not shoved straight to prod. Then
re-enable any flag you turned off.
Step 4 — Close out
- Update the status thread: "resolved," with a one-line cause.
- Write a short follow-up (timeline, root cause, what stopped it, prevention). If it revealed a new failure mode, add it to your team's runbook or gate so it can't recur silently.
- File the prevention work as a ticket.
Incident checklist
[ ] Triaged: blast radius + most-recent-change identified; status thread open
[ ] Bleeding stopped via the SAFE path (rollback / revert / flag-off / scale) — service confirmed healthy
[ ] No destructive "fixes" run (no scale-to-zero, destructive infra apply, force-delete-secret, reflex migration)
[ ] Root cause found from logs/alerts and reproduced off-prod
[ ] Real fix shipped through the normal gate/review/deploy pipeline; flags restored
[ ] Status thread resolved; follow-up written; prevention ticket filed
Anti-patterns
- Fixing forward a cause you don't understand instead of rolling back to known-good.
- A panicked scale-to-zero or destructive apply that deepens the outage.
- Silent recovery with no status thread and no follow-up — the next person relearns it.
- Shipping the real fix straight to prod, skipping review because "it's an incident."