References: Weaveworks Progressive Delivery, Argo Rollouts, Flagger, LaunchDarkly Feature Flags, Google SRE Release Engineering.
Applicable Scenarios
- Production release plan design
- Canary / blue-green / canary selection
- Rollback plan design
- Feature Flag management
- Database migration and code release coordination
- Multi-environment release process
Core Principles
1. 渐进式发布(Progressive Delivery)
不一次性全量,逐步放量
-
Observability-driven At each stage, look at metrics before deciding to continue / roll back
-
Rollback < 5 minutes Every release must be able to roll back quickly
-
Separate code and configuration Feature Flag controls feature toggles
-
Database before code Schema-compatible changes go first, code follows
-
Automation first Manual approval can exist, but execution must be automated
## 发布策略对比
| 策略 | 停机 | 风险 | 复杂度 | 适合 |
|---|---|---|---|---|
| **滚动更新** | 无 | 中 | 低 | K8s 默认 |
| **蓝绿部署** | 无 | 低 | 中 | 快速切换 |
| **金丝雀** | 无 | 极低 | 高 | 核心服务 |
| **灰度(百分比)** | 无 | 低 | 中 | 用户级控制 |
| **Feature Flag** | 无 | 极低 | 中 | 功能级控制 |
| **大爆炸** | 可能 | 高 | 低 | 内部工具 |
## 滚动更新(Rolling Update)
```yaml
# K8s 默认策略
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # 最多多 25% Pod
maxUnavailable: 25% # 最多少 25% Pod
流程:
旧 Pod: [A] [A] [A] [A]
→ 新建 1 个新 Pod: [A] [A] [A] [A] [B]
→ 删 1 个旧 Pod: [A] [A] [A] [B]
→ 新建 1 个新 Pod: [A] [A] [A] [B] [B]
→ ...
→ 全部替换: [B] [B] [B] [B]
优点:零停机、简单
缺点:新旧版本共存期间可能不兼容
Blue-Green Deployment
流程:
1. 当前生产(Blue)正常运行
2. 部署新版本到 Green 环境
3. 验证 Green(健康检查 + 冒烟测试)
4. 切换流量:Blue → Green
5. 观察 Green
6. 确认无问题后销毁 Blue
7. 出问题:切回 Blue(秒级)
优点:切换快、回滚快
缺点:需要双倍资源
# K8s 蓝绿(用 Service selector 切换)
apiVersion: v1
kind: Service
metadata:
name: my-app
spec:
selector:
app: my-app
version: green # 切换时改为 blue
Canary
流程:
1. 部署新版本到 1 个 Pod(5% 流量)
2. 监控 5 分钟(错误率 / P99)
3. 指标正常 → 扩到 50%
4. 监控 10 分钟
5. 指标正常 → 100%
6. 指标异常 → 自动回滚到 0%
工具:
- Argo Rollouts
- Flagger
- Istio + 流量分配
# Argo Rollouts 金丝雀
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
canary:
steps:
- setWeight: 5
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 10m }
- setWeight: 100
analysis:
templates:
- templateName: success-rate
startingStep: 1
Feature Flag
用途:
- 功能开关(不发版就能开关功能)
- 灰度(按用户 / 租户 / 百分比)
- A/B 测试
- Kill Switch(紧急关闭)
工具:
- LaunchDarkly(SaaS)
- Unleash(开源)
- Flagsmith(开源)
- GrowthBook(开源 + A/B)
- 自建(Redis + 配置)
代码示例:
if (featureFlags.isEnabled('new-checkout', { userId })) {
return <NewCheckout />;
}
return <OldCheckout />;
Rollback Plan
Three-Layer Rollback
1. 代码回滚(最快,< 1 min)
- K8s: kubectl rollout undo
- Vercel: 一键回滚
- Docker: 切回旧镜像 tag
2. 配置回滚(< 5 min)
- Feature Flag 关闭
- 环境变量回退
- ConfigMap 回退
3. 数据回滚(最慢,分钟~小时)
- 数据库 PITR
- 备份恢复
- 补偿脚本
Rollback Trigger Conditions
自动回滚(推荐):
- 错误率 > 5% 持续 2 分钟
- P99 > 2x 基线持续 5 分钟
- 健康检查失败 > 3 次
手动回滚:
- 业务指标异常(订单数骤降)
- 用户投诉激增
- 安全漏洞发现
K8s Rollback
# 查看历史
kubectl rollout history deployment/my-app
# 回滚到上一版本
kubectl rollout undo deployment/my-app
# 回滚到指定版本
kubectl rollout undo deployment/my-app --to-revision=3
# 查看状态
kubectl rollout status deployment/my-app
Release Process Template
1. 代码合并到 main
↓
2. CI 跑通(lint + test + build + scan)
↓
3. 构建镜像(tag = git sha)
↓
4. 部署到 staging
↓
5. staging 验证(自动 + 手动)
↓
6. 审批(如需)
↓
7. 部署到 production(金丝雀 5%)
↓
8. 监控 5 分钟
↓
9. 扩到 50% → 监控 10 分钟
↓
10. 扩到 100%
↓
11. 观察 1 小时
↓
12. 完成(或回滚)
Database and Code Coordination
原则:Schema 变更先于代码变更
发布顺序:
1. 数据库兼容变更(Expand)
2. 部署新代码(读写新字段)
3. 回填数据
4. 数据库清理变更(Contract)
详见 database-engineer/migration-rollout skill
Supporting Templates
templates/release-plan-template.md— Release plan + strategy + rollback + monitoring
Quality Self-Check
□ 发布策略选定(滚动 / 蓝绿 / 金丝雀)
□ 回滚方案可执行(< 5 min)
□ 回滚触发条件明确
□ 监控指标覆盖
□ staging 验证通过
□ 数据库变更先于代码
□ Feature Flag(如需)
□ 审批流程(如需)
□ 发布后观察期
□ 文档化(发布手册)
Common Pitfalls
- No rollback plan——panic during incidents
- Full release——affects all users at once
- Ramping up without monitoring——issues spread
- Code before database——new code cannot read new fields
- Rollback too slow——> 30 minutes
- Feature Flags not cleaned up——a pile of if/else in the code
- Skipping staging——only discovered in production
- Using latest for image tag——don't know which version to roll back to
- No automation——manual deployment errors
- Alerts not configured——don't know when problems occur after release
Collaboration with Other Skills
上游:
ci-cd-pipeline → 构建产物
containerization → 镜像
kubernetes-orchestration → 部署目标
下游:
monitoring-alerting → 发布后监控
sre-operations 工作流 → 事故响应
database-engineer → 迁移协调