maintain-infrastructure-observability

Observability — Observability

Entry/Exit

Entry: Before service launch, production issue investigation, monitoring configuration
Exit: Working monitoring Dashboard + alert rules
Points to: Continuous monitoring after launch, enter on anomaly verify-workflow-debug
Prerequisite loading: CANON.md
Output path: Enter on anomaly verify-workflow-debug; after launch enter ship-workflow-canary

When not to use

Not yet at the launch or production diagnosis stage, only fixing features locally
Only a single test run or log check, no need for long-term metrics/alerts
When root cause is unknown and there is no observable data, first enter verify-workflow-debug

Three Pillars

Metrics  → 数字：知道"有问题"          (WHAT)
Logging  → 事件：知道"具体是什么问题"    (WHAT + CONTEXT)
Tracing  → 请求链路：知道"问题在哪层"    (WHERE)

Metrics — Symptom Detection

RED method (service endpoints):

Rate — Requests per second
Errors — Failure rate
Duration — P50/P95/P99 latency

USE method (infrastructure/resources):

Utilization — CPU, memory, disk usage
Saturation — Queue depth, connection pool wait
Errors — Hardware errors, network packet loss

Logging — Context

// Bad: 信息不足
console.log('Task created');

// Good: 结构化 + 上下文
logger.info('task.created', {
  taskId: task.id,
  userId: req.user.id,
  title: task.title,
  duration: Date.now() - start,
});

Log levels:

Level	Purpose	Production environment
ERROR	Issues requiring manual intervention	Always on
WARN	Needs attention but can auto-recover	Always on
INFO	Key business events	Always on
DEBUG	Detailed diagnostic information	Enable on demand
TRACE	Line-by-line tracing	Never enable (too detailed)

Tracing — Request Chain

请求进入 → API Gateway(50ms) → Service(200ms) → DB Query(180ms) → 返回
                                          └→ Cache Read(2ms)

Distributed tracing must be used when spanning 3 or more services. Trace ID is passed from start to end (not rewritten).

Alert Design

告警的黄金信号:
├── 错误率突升（baseline 对比）→ P1
├── P95 延迟突升（baseline 对比）→ P1
├── 健康检查失败 → P0
├── 磁盘/内存使用率 > 90% → P2
├── SSL 证书即将过期（< 7 天）→ P2
└── 队列积压 > 阈值 → P2

Alert design principles:

Alert = symptom, not cause. "Database connection pool is full" is a symptom (alert), "N+1 queries cause too many connections" is the cause (debugging)
Alerts must have a Runbook link (documentation for "what to do")
Alert thresholds must have sufficient margin — P95 > 1s may trigger too frequently, P95 > 5s is a true anomaly

Health Check Endpoint

GET /health        → 公开。返回 200 或 503。负载均衡器用。
GET /health/ready  → 内部。检查 DB 连接、Redis 连接、关键依赖。
GET /health/live   → 内部。进程存活检查（最简单，仅返回 200）。

Dashboard Design

Layered by audience:

工程师 Dashboard:
├── 请求量 + 错误率 + 延迟（按端点）
├── 依赖健康（DB、Redis、Queue、外部 API）
├── 基础设施（CPU、内存、磁盘、网络）
└── 最近部署标记（竖线）

业务 Dashboard:
├── 用户操作量（注册、登录、关键操作）
├── 转化率（漏斗每步）
└── 支付成功率

Common Sayings

Saying	Reality	Consequence
"console.log is enough"	No structure, no context, no levels. Logging frameworks (pino/winston/logrus) cost very little and yield great benefits.	Production issue localization time ×3-5; cannot automatically aggregate alerts, rely on manually digging through logs each time
"Too many metrics to look at"	Alerts don't rely on humans watching. Set thresholds + automatic alerts + Runbook. Humans only intervene when alerts trigger.	Key anomalies missed → users discover first → SLA loss; manual monitoring cost is unsustainable
"Tracing is too expensive, skip it"	Mandatory when spanning more than 3 services. Without tracing, locating slow requests = troubleshooting each service one by one = costs more.	Cross-service slow request troubleshooting time ×5-10; MTTR rises from minutes to hours
"Production issues are only known when users report them"	This means you have no alerting. Users are doing monitoring for you.	Each user report = 1-2 hours of trust loss + brand damage; proactive alerting can compress response to 5 minutes

Red Flags — STOP

Production code uses console.log instead of structured logging
Log records contain passwords, tokens, PII (personally identifiable information)
No /health endpoint or /health always returns 200 (even if the DB is down)
Critical paths lack tracing (black box when spanning multiple services)
Alerts have no Runbook link (alert fires → no idea what to do)
Critical errors are caught but not logged (catch (e) {} silently swallowed)

Validation Failure Handling

Validation Item	Failure Symptom	Handling Method
Structured logging missing	Critical paths use console.log	Replace with logger.info/error, add context fields
Health check endpoint missing	No /health or always returns 200	Implement /health + /health/ready, check dependency connectivity
Core metrics not visible	Dashboard has no request volume/error rate/latency	Add RED metrics to Dashboard, configure auto-refresh
Alerts have no Runbook	No one knows what to do after an alert fires	Write a Runbook for each alert, link it to the alert configuration
Trace ID not propagated	Cross-service requests have no correlated trace_id	Inject trace_id into request headers, propagate and log it between services

Good and Bad Examples

Good: structured logging + alerting + Runbook

logger.info('task.created', {
  taskId: task.id,
  userId: req.user.id,
  duration: Date.now() - start,
});
// 告警: error_rate > 1% → P1, Runbook: /ops/runbooks/task-error-spike

Bad: console.log + no alerting

console.log('Task created');  // 无上下文、无级别、无法聚合
// 无告警 → 用户先发现生产问题

Output Template

可观测性配置完成：

Dashboard:
  - [端点] 请求量 + 错误率 + P95延迟
  - [依赖] DB / Redis / Queue 健康
  - [基础设施] CPU / 内存 / 磁盘

告警:
  - P0: 健康检查失败 → Runbook: <url>
  - P1: 错误率 > 1% → Runbook: <url>
  - P1: P95延迟 > 5s → Runbook: <url>
  - P2: 磁盘 > 90% → Runbook: <url>

健康检查:
  - GET /health → 公开 (负载均衡器)
  - GET /health/ready → 内部 (DB + Redis + 关键依赖)

追踪:
  - trace_id 传递: [是/否/不适用(单服务)]
  - 跨服务数: [N]

Validation Checklist

Structured logging records on critical business paths
/health + /health/ready endpoint exists
Core metrics are visible on the Dashboard
Critical alerts are configured (error rate, latency, health checks)
Each alert has a Runbook
Distributed trace IDs are propagated between services (if spanning 3+ services)
Logs do not contain secrets or PII