Applicable Scenarios
- SLO / SLI definition and review
- Error budget management
- Chaos Engineering (fault injection)
- Disaster recovery drills
- Reliability review
Core Principles
1. SLO 是合同
与 PM / 业务方对齐的可靠性目标
2. 错误预算是杠杆
预算充足 → 可以快速发布
预算耗尽 → 冻结发布,专注稳定
-
Proactively inject faults Don't wait for a production incident to discover weaknesses
-
100% availability is the wrong goal Pursuing 100% = no releases = no innovation
## SLO 框架
```text
SLI(指标):
- 可用性 = 成功请求 / 总请求
- 延迟 = P99 < 500ms 的比例
- 正确性 = 正确响应 / 总响应
SLO(目标):
- 可用性 99.9%(月 43 分钟不可用)
- 延迟 99% 请求 P99 < 500ms
错误预算:
- 月预算 = 1 - SLO = 0.1%
- 消耗 > 50% → 警告
- 消耗 > 80% → 冻结发布
Chaos Engineering
原则:
1. 定义稳态(正常指标)
2. 假设稳态会持续
3. 注入故障
4. 观察是否偏离稳态
5. 修复发现的弱点
常见实验:
- 杀 Pod(K8s)
- 网络延迟注入
- 磁盘满
- 第三方超时
- DNS 故障
- 数据库主从切换
工具:
- Chaos Monkey(Netflix)
- Litmus Chaos(K8s)
- Gremlin(SaaS)
- Chaos Mesh(K8s)
- toxiproxy(网络故障)
Supporting Templates
templates/slo-definition-template.md— SLO definition + error budget + Chaos experiment
Quality Self-Check
□ SLO 与业务方对齐
□ 错误预算跟踪
□ 预算耗尽有动作
□ Chaos 实验定期
□ 灾备演练(季度)
□ 弱点修复跟踪
Common Pitfalls
- SLO 100%——unrealistic
- SLO not aligned with the business——technical self-indulgence
- Error budget not tracked——exists in name only
- Not doing Chaos——weaknesses only discovered in production
- Afraid to do Chaos in production——start with staging
Collaboration with other skills
上游:
devops-engineer monitoring-alerting → SLI 数据
下游:
postmortem → SLO 违反时复盘
capacity-planning → 可靠性影响容量
incident-response → SLO 违反触发响应