monitoring-alerting

Monitoring & Alerting

References: Google "Site Reliability Engineering", Brendan Gregg USE Method, Prometheus official, Grafana Labs, PagerDuty Incident Response.

Applicable Scenarios

Monitoring system setup (Prometheus + Grafana)
Log collection (Loki / ELK)
Alert rule design
SLO / SLI definition
Dashboard design
On-call configuration

Core Principles

1. 四大黄金信号（Google SRE）
   Latency / Traffic / Errors / Saturation

USE Method (Brendan Gregg) For each resource: Utilization / Saturation / Errors
Alerts are based on SLO, not on thresholds "Error budget consumption > 50%" is more meaningful than "CPU > 80%"
Alerts are actionable Receive an alert -> know what to do Non-actionable alerts = noise
Tiered alerts P1 (immediate) / P2 (1 hour) / P3 (working hours) / P4 (next week)
No alert fatigue Better to have fewer alerts than so many that they are ignored


## 监控栈选型

| 组件 | 开源 | SaaS |
|---|---|---|
| **指标** | Prometheus + Grafana | Datadog / New Relic |
| **日志** | Loki / ELK | Datadog Logs / Splunk |
| **追踪** | Jaeger / Tempo | Datadog APM / Honeycomb |
| **告警** | Alertmanager | PagerDuty / OpsGenie |
| **Uptime** | Blackbox Exporter | UptimeRobot / Better Stack |
| **错误** | Sentry（自建） | Sentry Cloud |
| **APM** | OpenTelemetry | Datadog / Dynatrace |

## 四大黄金信号

```yaml
# Prometheus 规则示例

# 1. Latency（延迟）
- record: http_request_duration_seconds:p99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# 2. Traffic（流量）
- record: http_requests_per_second
  expr: rate(http_requests_total[5m])

# 3. Errors（错误率）
- record: http_error_rate
  expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# 4. Saturation（饱和度）
- record: cpu_saturation
  expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

SLO / SLI Definition

SLI（Service Level Indicator）：
  - 可用性：成功请求 / 总请求
  - 延迟：P99 < 500ms 的请求比例
  - 吞吐：每秒处理请求数

SLO（Service Level Objective）：
  - 可用性 SLO：99.9%（每月允许 43 分钟不可用）
  - 延迟 SLO：99% 请求 P99 < 500ms
  - 错误 SLO：错误率 < 0.1%

错误预算（Error Budget）：
  - 月预算 = 1 - SLO = 0.1% = 43 分钟
  - 消耗 > 50% → 冻结发布
  - 消耗 > 80% → 专注稳定性

# Prometheus SLO 告警（Burn Rate）
groups:
  - name: slo-alerts
    rules:
      # 1 小时 burn rate > 14.4（快速消耗）→ P1
      - alert: HighErrorBudgetBurn_1h
        expr: |
          (
            rate(http_requests_total{status=~"5.."}[1h])
            / rate(http_requests_total[1h])
          ) > 14.4 * 0.001
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error budget burning fast (1h window)"
      
      # 6 小时 burn rate > 6 → P2
      - alert: HighErrorBudgetBurn_6h
        expr: |
          (
            rate(http_requests_total{status=~"5.."}[6h])
            / rate(http_requests_total[6h])
          ) > 6 * 0.001
        for: 5m
        labels:
          severity: warning

Alert Rule Design

Severity Levels

Level	Meaning	Response Time	Notification Method
P1 Critical	Service unavailable / data loss	Immediate	Phone + Slack
P2 High	Feature degradation / performance regression	1 hour	Slack + email
P3 Medium	Non-urgent but needs attention	Working hours	Slack
P4 Low	Informational	Next week	Email

Good Alerts vs Bad Alerts

好告警：
  ✅ "订单创建错误率 > 5% 持续 2 分钟"
  ✅ "错误预算消耗 > 50%"
  ✅ "数据库连接池使用 > 90%"

坏告警：
  ❌ "CPU > 80%"（可能正常）
  ❌ "磁盘 > 70%"（太早）
  ❌ "某个 Pod 重启了"（可能正常）
  ❌ "日志有 ERROR"（太多噪音）

Alert Templates

- alert: HighErrorRate
  expr: http_error_rate > 0.05
  for: 2m
  labels:
    severity: critical
    team: backend
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"
    description: "Error rate > 5% for 2 minutes"
    runbook: "https://wiki.example.com/runbooks/high-error-rate"
    dashboard: "https://grafana.example.com/d/xxx"

Grafana Dashboard Design

5 Recommended Dashboards

1. 服务总览（Golden Signals）
   - QPS / 错误率 / P50 / P99 / 饱和度
   - 按服务分组

2. 业务指标
   - 订单数 / 支付成功率 / 注册数
   - 按时间对比（vs 昨天 / 上周）

3. 基础设施
   - CPU / Memory / Disk / Network
   - 按节点 / Pod 分组

4. 数据库
   - 连接数 / 慢查询 / 锁等待 / 复制延迟

5. 发布追踪
   - 部署时间线
   - 发布前后指标对比

Prometheus + Grafana Deployment

# docker-compose（快速启动）
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:8080']
    metrics_path: '/metrics'

Log Collection (Loki)

# Loki + Promtail
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail.yml:/etc/promtail/config.yml

日志规范：
  - JSON 格式（结构化）
  - 含 trace_id / service / level / timestamp
  - 不输出 PII / Token
  - 保留期：7~30 天

On-call Configuration

轮值：
  - 每周轮换
  - 主 + 副（backup）
  - 交接文档

升级路径：
  P1：5 分钟无响应 → 升级到副
  P1：15 分钟无响应 → 升级到 Tech Lead
  P1：30 分钟无响应 → 升级到 CTO

工具：
  - PagerDuty / OpsGenie / Grafana OnCall
  - Slack 集成
  - 电话 + 短信

Supporting Templates

templates/monitoring-setup-template.md — Monitoring configuration + SLO + alert rules + dashboards
templates/alert-rule-template.md — Alert rules and release gates

Quality Self-check

□ 四大黄金信号覆盖
□ SLO / SLI 定义
□ 告警基于 SLO（不基于阈值）
□ 告警分级（P1~P4）
□ 告警可操作（有 Runbook）
□ 不 alert fatigue
□ 仪表盘覆盖（5 个）
□ 日志结构化 + 集中
□ On-call 配置
□ 升级路径
□ 错误预算跟踪
□ 发布后监控

Common Pitfalls

CPU > 80% alert——may be normal; you should look at SLO
Too many alerts——alert fatigue, important ones are ignored
Alerts are not actionable——when received, you do not know what to do
No severity levels——all alerts are P1
No Runbook——woken up in the middle of the night and do not know how to fix it
Logs are not structured——difficult to search
Business metrics are not monitored——only watching technical metrics
SLO is not defined——do not know what counts as "normal"
No error budget——always firefighting
On-call does not rotate——one person carries it alone

Collaboration with Other Skills

上游：
  ci-cd-pipeline → 部署后触发监控
  release-strategy → 发布后观察
  kubernetes-orchestration → K8s 指标

下游：
  sre-operations 工作流 → 事故响应
  backend-engineer observability → 应用层指标

About this skill

Applicable Scenarios

Core Principles

SLO / SLI Definition

Alert Rule Design

Severity Levels

Good Alerts vs Bad Alerts

Alert Templates

Grafana Dashboard Design

5 Recommended Dashboards

Prometheus + Grafana Deployment

Log Collection (Loki)

On-call Configuration

Supporting Templates

Quality Self-check

Common Pitfalls

Collaboration with Other Skills