ship-infrastructure-deploy

Deploy — Deployment Management

Entry/Exit

• Entry: preparing to release,ship-workflow-ship process's deployment phase
• Exit: successful deployment + post-release verification + rollback playbook
• Points to: deployment complete → ship-workflow-canary Monitoring
• Prerequisite loading: CANON.md、ship-infrastructure-ci-cd/SKILL.md
• Output path: deployment record → ship-workflow-canary Canary Monitoring

When not to use

• Has not yet passed the review/ship gate, cannot enter actual deployment
• Only preparing a release checklist or exporting non-software artifacts
• After deployment, only health monitoring or regression observation is needed

Iron Law

Pre-Launch Checklist

Must confirm item by item before release:

Code quality

• All tests pass (unit + integration + E2E)
• CI/CD last build is green
• Code review complete (no unresolved Critical/Important findings)
• No merge conflicts

Security

• npm audit(or equivalent) no critical/high
• No secrets/credentials in code
• Security headers configured (CSP, HSTS, CORS allowlist)
• Rate limiting configured

Performance

• Lighthouse / Core Web Vitals meet targets
• Database migration verified in staging
• No N+1 queries

Infrastructure

• Migration scripts have UP + DOWN (all reversible)
• Environment variables set in target environment
• SSL certificate valid (not expired)
• Database backup complete

Documentation

• CHANGELOG updated
• API docs updated (if there are API changes)
• Relevant sections in README updated

Feature Flag Strategy

Decouple deployment (releasing code) and release (enabling functionality).

Flag 生命周期:
  created ──→ enabled (0% → 10% → 50% → 100%) ──→ removed (代码清理)
                │
                └── 出问题 → 立即 0%（关闭 flag）—— 不回滚部署

Rules:

1. Every flag has an owner (who created it, who is responsible for removing it)
2. Every flag has an expiration date (automatically removed or reminded when overdue)
3. Flag code must be concise — removing the flag deletes only one line, no refactoring
4. When the flag is off, the user experience is equivalent to the flag not existing — not a degraded experience

Phased Rollout

Phase 1: Staging     → 验证所有功能 + 迁移
Phase 2: 金丝雀 (5%)  → 观察 15 分钟。关注核心指标。
Phase 3: 扩大 (25%)   → 观察 30 分钟。比较 canary vs baseline。
Phase 4: 半数 (50%)   → 观察 1 小时。
Phase 5: 全量 (100%)  → 持续观察 24 小时。

Advancement conditions: The core metrics of each phase are stable (error rate, latency, success rate not worse than baseline). Any anomaly → pause advancement → investigate → fix or roll back.

Rollback Triggers

Trigger condition	Action
Error rate rises significantly (> 2x baseline)	Immediately roll back or turn off feature flag
P50/P95 latency rises significantly (> 1.5x baseline)	Immediately roll back
Critical user flows broken (500 on critical pages)	Immediately roll back
Database migration fails or data is inconsistent	Roll back migration + roll back deployment
Security vulnerability discovered	Immediately roll back, release again after subsequent security audit

Rollback Playbook

## Rollback Plan

### Trigger
- 错误率 > X% for 5 min
- P95 latency > Yms for 5 min
- 关键页面 500 rate > Z%

### Steps
1. [command to revert deployment]
2. [command to rollback migration: npx db:migrate:down]
3. [command to close feature flag: curl -X POST /api/flags/X/disable]

### RTO (Recovery Time Objective)
< 5 min from trigger

### RPO (Recovery Point Objective)
Zero data loss

### Owner
[name / oncall rotation]

Post-Launch Verification

1. 错误追踪 → 实时查看错误率
2. 关键页面 → 手动验证核心用户流程
3. 性能 Dashboard → 对比上线前后
4. 健康检查端点 → /health 返回 OK
5. SSL → 证书未过期
6. 迁移 → 验证数据完整性

Common Excuses

Excuse	Reality	Consequence
"It's just a small change, no need to phase it"	Small changes can also cause big problems. A canary deployment takes 15 minutes, which is cheaper than a full rollback taking 2 hours.	Small change not phased → affects all users → when problems occur, all users are affected simultaneously → rollback 2 hours vs canary detection in 15 minutes.
"Write the rollback playbook later"	There's no time to "write it later" when problems occur during launch. Rollback must be muscle memory.	No rollback playbook → panicked guessing of rollback steps when problems occur → average rollback time > 30 minutes vs with a playbook < 5 minutes.
"Feature flags are too complex"	One if/else condition. Simpler than an emergency rollback deployment.	No flag → when problems occur, can only roll back the entire deployment → all users affected vs flag off → < 1 minute recovery, only affects new feature users.
"Launch on Friday afternoon first"	Launching Friday afternoon = if problems occur, no one is around on the weekend to maintain it. Launch Tuesday/Wednesday morning.	Friday launch → problems occur → no response on weekend → users affected for 48-72 hours vs Tuesday launch → problems occur → immediate response.

Red Flags — STOP

• About to launch without a rollback plan
• Feature flag has no expiration date and owner
• All traffic switched to the new version at once (no canary)
• Database migration has UP but no DOWN
• Not immediately checking Dashboard/error rate after launch
• Secrets/environment variables not configured in production

Verification Checklist

• All Pre-Launch checklist items passed
• Rollback playbook is clear and executable
• Feature flags have owner + expiration date
• Canary deployment verification passed
• Post-launch core metrics show no significant degradation vs baseline
• Manually verify critical user flows after launch
• CHANGELOG has been updated

Handling Verification Failures

Failure Scenario	Handling Method
Pre-Launch checks did not all pass	STOP. Fill in the missing items before launching. "Mostly passed" is not acceptable.
Canary phase metrics degrade	Pause rollout → investigate root cause → after fixing, re-canary or roll back. Do not force an increase in traffic percentage.
Database migration fails	Roll back the migration (DOWN script) + roll back the deployment. Fix the migration script and try again.
Critical user flow breaks after launch	Immediately roll back or turn off the feature flag. Do not debug first and then decide—restore service first, then investigate.
Rollback playbook is not executable	STOP. Rewrite the rollback playbook, ensuring each step has specific commands and verification methods. No playbook, no launch.

Good and Bad Examples

Good — Phased launch with a rollback playbook

Phase 1: Staging → 全部验证通过 ✅
Phase 2: 金丝雀 5% → 15 分钟观察 → 错误率正常 ✅
Phase 3: 25% → 30 分钟 → P95 无恶化 ✅
Phase 4: 全量 → 持续监控

回滚剧本:
1. kubectl rollout undo deployment/app
2. npx db:migrate:down
3. curl -X POST /api/flags/notification/disable
RTO: < 5 min

Bad — Big bang launch with no rollback

（直接全量切换，不分阶段，无回滚剧本）

→ 问题: 全量切换 → 5% 用户遇到 500 也影响全部
→ 问题: 无回滚剧本 → 出问题后慌乱猜测步骤 → 回滚 > 30 分钟
→ 问题: 无 feature flag → 不能快速关闭新功能 → 必须回滚整个部署

Output Template

### Deploy 交付记录 — <feature-name>

**部署策略**: [staging → canary → staged rollout / blue-green / rolling update]
**回滚剧本**: [具体命令] — RTO: < 5 min
**Feature Flags**: [flag名 / owner / 过期日期]

**Pre-Launch 检查**:
- 代码质量: [全部通过 / 具体未通过项]
- 安全: [全部通过 / 具体未通过项]
- 性能: [全部通过 / 具体未通过项]
- 基础设施: [全部通过 / 具体未通过项]
- 文档: [全部通过 / 具体未通过项]

**分阶段上线记录**:
| Phase | 流量比例 | 观察时长 | 核心指标 | 结果 |
|-------|---------|---------|---------|------|
| Staging | 100% | [时长] | [指标] | ✅ / ❌ |
| Canary | 5% | 15min | [指标] | ✅ / ❌ |

**健康验证**: [端点状态 + 响应时间 — PASS/FAIL]