SuperAPI AI Directory

reliability-engineering

General & other

View repository

Updated 10-16 09:31

About this skill

Reliability Engineering

Applicable Scenarios

  • SLO / SLI definition and review
  • Error budget management
  • Chaos Engineering (fault injection)
  • Disaster recovery drills
  • Reliability review

Core Principles

1. SLO 是合同
   与 PM / 业务方对齐的可靠性目标

2. 错误预算是杠杆
   预算充足 → 可以快速发布
   预算耗尽 → 冻结发布,专注稳定
  1. Proactively inject faults Don't wait for a production incident to discover weaknesses

  2. 100% availability is the wrong goal Pursuing 100% = no releases = no innovation


## SLO 框架

```text
SLI(指标):
  - 可用性 = 成功请求 / 总请求
  - 延迟 = P99 < 500ms 的比例
  - 正确性 = 正确响应 / 总响应

SLO(目标):
  - 可用性 99.9%(月 43 分钟不可用)
  - 延迟 99% 请求 P99 < 500ms

错误预算:
  - 月预算 = 1 - SLO = 0.1%
  - 消耗 > 50% → 警告
  - 消耗 > 80% → 冻结发布

Chaos Engineering

原则:
  1. 定义稳态(正常指标)
  2. 假设稳态会持续
  3. 注入故障
  4. 观察是否偏离稳态
  5. 修复发现的弱点

常见实验:
  - 杀 Pod(K8s)
  - 网络延迟注入
  - 磁盘满
  - 第三方超时
  - DNS 故障
  - 数据库主从切换

工具:
  - Chaos Monkey(Netflix)
  - Litmus Chaos(K8s)
  - Gremlin(SaaS)
  - Chaos Mesh(K8s)
  - toxiproxy(网络故障)

Supporting Templates

  • templates/slo-definition-template.md — SLO definition + error budget + Chaos experiment

Quality Self-Check

□ SLO 与业务方对齐
□ 错误预算跟踪
□ 预算耗尽有动作
□ Chaos 实验定期
□ 灾备演练(季度)
□ 弱点修复跟踪

Common Pitfalls

  1. SLO 100%——unrealistic
  2. SLO not aligned with the business——technical self-indulgence
  3. Error budget not tracked——exists in name only
  4. Not doing Chaos——weaknesses only discovered in production
  5. Afraid to do Chaos in production——start with staging

Collaboration with other skills

上游:
  devops-engineer monitoring-alerting → SLI 数据

下游:
  postmortem → SLO 违反时复盘
  capacity-planning → 可靠性影响容量
  incident-response → SLO 违反触发响应