ship-workflow-canary

Canary — Post-deployment health monitoring

Entry/Exit

• Entry: Code that has passed review and been deployed
• Exit: docs/features/<name>/06-canary-report.md
• Output path: 06-canary-report.md → reflect-team-retro
• Points to: Once monitoring is stable, recommend /retro
• Preload: CANON.md

When not to use

• Code has not been deployed yet, still before review/ship/deploy
• No production or observable environment available for verification
• Just a local smoke test or one-off manual check

Iron Law

Flow

Phase 1: Baseline collection

Collect a health baseline before deployment (or immediately after deployment):

# 对每个关键端点采集基线
for endpoint in /health /api/status /api/readyz; do
  echo "--- $endpoint ---"
  start=$(date +%s%N)
  status=$(curl -s -o /dev/null -w '%{http_code}' "https://${HOST}${endpoint}")
  end=$(date +%s%N)
  elapsed=$(( (end - start) / 1000000 ))
  body_hash=$(curl -s "https://${HOST}${endpoint}" | shasum -a 256 | cut -d' ' -f1)
  echo "{\"endpoint\":\"$endpoint\",\"status\":$status,\"response_ms\":$elapsed,\"body_hash\":\"$body_hash\"}"
done

Output baseline.json：

{
  "collected_at": "2025-01-15T10:00:00Z",
  "endpoints": [
    {
      "endpoint": "/health",
      "status_code": 200,
      "response_ms": 45,
      "body_hash": "a1b2c3..."
    }
  ]
}

Baseline rules:

• Collect at least 3 samples per endpoint, take the median
• Record status code + response time + body hash
• Baseline collection failure = blocked. Cannot start monitoring without knowing "what normal looks like"

Phase 2: Continuous monitoring

Poll endpoints every 60 seconds and compare against the baseline:

# 单次检查脚本
check_endpoint() {
  local endpoint=$1 baseline_status=$2 baseline_ms=$3
  local start status elapsed

  start=$(date +%s%N)
  status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "https://${HOST}${endpoint}")
  end=$(date +%s%N)
  elapsed=$(( (end - start) / 1000000 ))

  # 分级判定
  if [ "$status" = "000" ]; then
    echo "CRITICAL|$endpoint|不可达"
  elif [ "$status" != "$baseline_status" ] && [ "$status" -ge 500 ]; then
    echo "CRITICAL|$endpoint|新5xx:$status"
  elif [ "$status" != "$baseline_status" ] && [ "$status" -ge 400 ]; then
    echo "HIGH|$endpoint|新错误码:$status(基线:$baseline_status)"
  elif [ "$elapsed" -gt $((baseline_ms * 2)) ]; then
    echo "MEDIUM|$endpoint|响应时间${elapsed}ms>2x基线${baseline_ms}ms"
  elif [ "$status" != "$baseline_status" ] && [ "$status" -ge 300 ]; then
    echo "LOW|$endpoint|新重定向:$status(基线:$baseline_status)"
  else
    echo "OK|$endpoint|status=$status time=${elapsed}ms"
  fi
}

Alert severity levels:

Level	Condition	Meaning
CRITICAL	Endpoint unreachable (curl timeout/connection refused)	Service is down
HIGH	New 4xx/5xx error code	Functionality broken
MEDIUM	Response time > 2x baseline	Performance degradation
LOW	New redirect	Possible configuration change

Debounce rules: Alert only after the same severity-level anomaly is detected 2+ times consecutively. A single anomaly is recorded as TRANSIENT, without alerting.

Monitoring duration:

• Small change: 30 minutes
• Standard change: 2 hours
• Major change (core path/database migration): 24 hours

Phase 3: Health report

After monitoring ends, generate an endpoint health status summary:

Status	Definition
HEALTHY	All checks OK, no TRANSIENT
DEGRADED	Has LOW/MEDIUM level anomalies, but no HIGH/CRITICAL
BROKEN	Has HIGH/CRITICAL level anomalies

Output docs/features/<name>/06-canary-report.md：

# Canary Report — <name>

## Canary Summary
- Owner:
- Date:
- Monitoring window:
- Change size: small / standard / major
- Result: HEALTHY / DEGRADED / BROKEN
- Decision: continue / pause / rollback
- Ship source: `docs/features/<name>/05-ship.md`

## Canary Scope
- Environment / region / ring:
- Canary population / traffic percentage:
- Version / commit / artifact:
- Critical user journeys:
- Out of scope:

## Baseline
- Baseline source:
- Baseline collected at:
- Endpoint count:
- Samples per endpoint:
- Baseline owner:

## Health Signals
| Signal | Source | Baseline | Current | Threshold | Status |
|--------|--------|----------|---------|-----------|--------|

## Endpoint Status

| Endpoint | Status | Baseline Response | Current / Average Response | Error Count | Notes |
|----------|--------|-------------------|----------------------------|-------------|-------|

## Analysis Policy
- Interval:
- Count / duration:
- Failure threshold:
- Debounce rule:
- Promotion rule:
- Rollback rule:

## Anomalies
| Time | Signal / Endpoint | Severity | Evidence | Action |
|------|-------------------|----------|----------|--------|

## Decision
- Final status: HEALTHY / DEGRADED / BROKEN
- Decision: continue / pause / rollback
- Decision reason:
- Decision owner:
- Decision time:

## Baseline Update
- Updated: yes / no
- Reason:
- Previous baseline archived at:

## Follow-up
- Owner:
- Tracking:
- Next check:
- Next command: `/retro` / continue monitoring / rollback / investigate

Phase 4: Baseline update

When all endpoints are HEALTHY, optionally update the baseline:

# 用最新数据覆盖 baseline.json
mv current-round.json baseline.json

Rules:

• Update only when all endpoints are HEALTHY
• Do not update when DEGRADED or BROKEN — the baseline represents "known good state"
• Before updating, save the old baseline as baseline-$(date +%Y%m%d).json

Verification evidence

Output or records must include:

• Input/source: The spec, plan, code, feedback, or release context that was read.
• Actions performed: The checks, generation, fixes, exports, or release steps actually completed.
• Verification results: Commands, review conclusions, artifact paths, screenshots, or manual confirmation.
• Blockers/rollback: Failed items, rollback paths, or issues requiring a human partner's decision.

Common claims

Claim	Reality	Consequence
"Once deployed, there's nothing to worry about"	Deployment is only half of going live. Monitoring in the first hour after going live determines whether users are affected.	Going live without monitoring: failures are discovered on average 15 minutes later through user complaints, affecting 100+ users, and repairing the loss of trust takes 1-2 weeks.
"No baseline needed, just look at the logs"	Logs tell you "what happened"; a baseline tells you "whether this is normal." Without a baseline, all data is noise.	Monitoring without a baseline: each alert requires 20-40 minutes of manual judgment to determine whether it is normal, extending MTTR from 5 minutes to 45+ minutes.
"A 60-second check is too frequent"	60 seconds can detect a failure within 2 minutes (after debouncing). It takes users an average of 15 minutes to discover a failure.	5-minute interval checks: failure detection delay increases from 2 minutes to 10+ minutes, affecting 50-200 active users per minute.
"We'll look when there's a problem"	By the time there's a problem, you've already lost users. Proactive monitoring shortens discovery time from "user complaints" to "minute-level."	Passively waiting for complaints: failure exposure is delayed by 30-60 minutes, customer churn increases by 3-5%, and each incident causes a direct loss of ¥5K-¥50K.
"As long as it's 200, it's fine"	200 but response time rises from 50ms to 5000ms = users can no longer use it. Status code is not the whole of health.	Looking only at status code: a 5-second slow response is equivalent to a service outage in user experience, conversion rate drops by 20-40%, and 10% of users leave directly.

Red flags

• Starting monitoring without a baseline
• Checking only status code and not response time
• Alerting on a single anomaly (no debouncing)
• Declaring "stable" after monitoring for 5 minutes
• Baseline collection fails but deployment still continues
• BROKEN status but no rollback triggered
• No one on duty during monitoring (24-hour monitoring for major changes requires oncall)
• Critical business endpoints are not in the monitoring list

Good/Bad examples

Bad: only looking at status code

Canary 结果: 所有端点返回 200，部署成功。

Problem: no baseline comparison, no response time data, no anomaly details. 200 may be accompanied by a 10x response time regression.

Good: evidence-driven canary report

Canary Report — payment-v2

## 摘要
- 监控时长: 2 小时
- 检查次数: 120
- 结果: DEGRADED

## 端点状态
| 端点 | 状态 | 基线响应 | 平均响应 | 异常次数 |
|------|------|---------|---------|---------|
| /health | HEALTHY | 45ms | 48ms | 0 |
| /api/payments/process | DEGRADED | 320ms | 780ms | 8 (MEDIUM) |

## 异常详情
- 14:05 /api/payments/process 响应 780ms (2.4x 基线 320ms)
- 14:06 /api/payments/process 响应 810ms (2.5x 基线)
- 连续 8 次超过 2x 基线，防抖确认 MEDIUM

## 建议
- /health HEALTHY → 可更新基线
- /api/payments/process DEGRADED → 继续监控 30 分钟，排查支付服务慢查询
- 如 30 分钟后仍 DEGRADED → 建议回滚

Advantages: baseline comparison, quantified regression degree, debounce confirmation, clear next action.

Output template

Template starting point:templates/feature/06-canary-report.md

# <Feature Name> — Canary Report

## Canary Summary
## Canary Scope
## Baseline
## Health Signals
## Endpoint Status
## Analysis Policy
## Anomalies
## Decision
## Baseline Update
## Follow-up

Validation failure handling

Failure scenario	Handling method
Baseline collection fails (endpoint unreachable)	Block deployment, fix the endpoint or recollect after exclusion, and do not start monitoring without a baseline
2+ consecutive CRITICAL alerts	Immediately trigger rollback, execute according to the rollback plan, and notify the human partner
Response time >2x baseline for 10 consecutive minutes	Downgrade to DEGRADED, continue monitoring for 30 minutes; if degradation persists, recommend rollback
TRANSIENT occurs >5 times within 30 minutes of monitoring	Extend monitoring to 2 hours, investigate the root cause of the jitter, and do not declare HEALTHY
DEGRADED endpoints still remain at the end of monitoring	Do not update the baseline, mark the degraded endpoints in the output report, and recommend that the human partner decide whether to continue or roll back

Validation checklist

• baseline.json collected (median of at least 3 runs)
• Canary Scope clearly states environment, ring, traffic proportion, version, and key user paths
• Health Signals cover availability, latency, error rate, and necessary business signals
• Analysis Policy clearly states interval, count/duration, failure threshold, anti-flapping, promotion, and rollback rules
• All critical endpoints are in the monitoring list
• Anti-flapping mechanism is effective (alerts only after 2+ consecutive occurrences)
• Monitoring duration matches the scale of the change
• 06-canary-report.md has been generated
• Decision records continue / pause / rollback and owner
• Follow-up records next steps and tracking
• All endpoint statuses are HEALTHY (or DEGRADED/BROKEN has corresponding actions)
• HEALTHY endpoint baselines have been updated