observability-and-instrumentation

Observability and instrumentation

Overview

Code you cannot observe, you cannot operate. Observability means: using the telemetry emitted by code to answer "what is the system doing, and why" from the outside. Instrumentation is not an add-on after launch—it is written alongside feature code, just like tests. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to use

• Building any feature that will run in production
• Adding a new service, endpoint, background job, or external integration
• Diagnosing a production incident took too long ("we couldn't tell what happened")
• Creating or reviewing alerting rules
• Reviewing a PR that adds new I/O, retries, queues, or cross-service calls

Not applicable to:

• Diagnosing an ongoing failure—use the debugging-and-error-recovery skill (observability is exactly what makes this skill fast the next time it runs)
• Profiling and optimizing measured slowness—use performance-optimization skill
• Launch-day monitoring checklist and rollback triggers—see the shipping-and-launch skill; this skill covers the instrumentation layer that feeds them

Process

1. Define "working" first, then instrument

Telemetry without questions to support it is noise. Before adding any instrumentation, write down the 2–4 questions an on-call engineer would ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

If you cannot state these questions, you are not ready to instrument—you will log everything and learn nothing.

2. Choose the right signal for each question

Signal	Answers what	Cost characteristics	Example
Structured logs	"What happened in this specific case?"	Per event; grows with traffic	with provider error code payment_failed
Metrics	"Aggregated, how often / how fast?"	Fixed per series; cheap to query	p99 latency of provider calls
Trace	"Which segment across services is the time spent in?"	Per request; usually sampled	A slow checkout, broken down by hop

Rule of thumb: metrics tell youwhat wentwrong, traces tell you where the problem iswhere, and logs tell youwhy。

3. Structured logs

Log events, not prose. Each log line is a JSON object with a stable event name and machine-readable fields:

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels—use them appropriately and consistently:

Level	Meaning	On-call action
error	An invariant is broken; someone may need to act	Investigate
warn	Degraded but handled (retry succeeded, fallback used)	Watch the trend
info	Important business events (order completed, job finished)	None
debug	Diagnostic details	Off by default in production

The Correlation ID is mandatory. Generate (or accept) a request ID at the system boundary, and attach it to every log line, every span, and every outbound call. Without it, you cannot reconstruct a single request from interleaved logs:

// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

When multiple entry points write to the same log, tag the logs with the entry point. The Correlation ID identifies a run, but it cannot tell you which code path initiated it. The same job triggered once each by the scheduler, the replay endpoint, and a manual CLI run will produce mutually substitutable lines in the same sink, so line attribution can only rely on elimination—cross-reading the scheduler's history, the process table, and deployment logs—and this argument only holds if exactly those outer records still exist. Where the run starts, tag the entry point right next to the correlation ID, and have both fields propagate in the same way:

// One helper for every entry point: the run's own logger carries both fields.
// `entryPoint`, not `source` — ECS reserves `source.*` for network fields.
export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) =>
  logger.child({ entryPoint, requestId: runId });

// scheduler tick        -> runLog('scheduler', crypto.randomUUID())
// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id)
// CLI invocation        -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID())

These two fields must cross the same boundaries as the correlation ID—queue metadata, HTTP headers—otherwise the worker can only re-infer the entry point and rely on guesswork. A field that merely correlates with some entry point is only a clue, not attribution: anything that can trigger this job can reproduce it.

Never log secrets, tokens, passwords, or full PII. This is from security-and-hardening a hard rule of the skill—the telemetry pipeline is a classic data-leak path. Log fields by allowlist; do not log the entire request body.

4. Metrics

For request-driven services, instrument every endpoint and every external dependency with RED：Rate (requests/second),Errors (failure rate),Duration (latency histogram, not average). For resources (queues, connection pools, hosts), use USE：Utilization、Saturation、Errors。

As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (the same SDK and context as in step 5). The example below uses Prometheus's prom-client—a common backend choice, not the only choice; RED/USE and the cardinality rules are exactly the same under both paths.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],  // '2xx', not '200'
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels—those belong in logs and traces.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Never track averages, always track percentiles: averages hide the 1% of users with a terrible experience. Use histograms, read p50/p95/p99.

5. Distributed tracing

Use OpenTelemetry—it is the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:

// tracing.ts — must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Add manual spans only around meaningful internal units of work (for example applyDiscounts、chargeProvider), and attach attributes that on-call will use for filtering. Let context propagate across every async boundary—HTTP headers, queue message metadata—otherwise the trace dies at the break point. By default, do head-based sampling at a low rate; if your backend supports tail sampling, retain 100% of errors.

6. Alerting

Alert onsymptoms users can feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Cause-based alerts fire when nothing is wrong and miss the failures you didn't predict. Symptom-based alerts fire exactly when users are harmed, regardless of the cause.

Rules for every alert you create:

1. It must be actionable. If the response is "ignore it, it'll self-heal," delete the alert.
2. It links to a runbook—even just three lines: what it means, the first query to run, the escalation path.
3. It has a threshold and duration, based on an SLO or historical data, not guesswork.
4. Use only two severity levels:page(user-facing, act immediately) and ticket(degraded, act within the week). A third tier becomes noise and teaches people to ignore everything.

Writing Runbooks

Rule 2 above requires every alert to link to a runbook. A runbook's job is to answer three questions without requiring the reader to think: what is happening, what to check first, and who to contact if that doesn't solve it. Store it in docs/runbooks/, named after the alert.

Minimum viable runbook (three lines):

# Runbook: High Error Rate on /api/tasks
**Means:** DB connection pool likely exhausted, or a bad deploy.
**First check:** `SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';`
  — if count > pool limit, see Step 2. (Swap in the equivalent for your database.)
**Escalate to:** #db-oncall or engineering on-call rotation.

When it's worth expanding beyond three lines: Only add steps when the first check alone isn't enough to make a judgment. A five-step runbook covering the three most common causes is better than a twenty-step document covering every edge case that people then skim past.

Keep runbooks current. Make updating the runbook part of closing every incident that used it—stale runbooks build false confidence. If a step is wrong or missing, fix it before marking the incident as resolved.

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Before declaring done, trigger those paths and look at the actual output:

• Deliberately cause an error in staging → find it in the logs with requestId and confirm the fields are structured (not [object Object]）
• Send test traffic → confirm the metric series appears with the expected labels and reasonable values
• Follow a request across services in the tracing UI → no broken spans
• Fire each new alert once (temporarily lower the threshold) → confirm it reaches the right channel and the runbook link works

Common rationalizations

Rationalization	Reality
"I'll add logging once it works"	"Later" becomes "after the first incident," and that's the most expensive moment to discover you're blind. Instrument as you build.
"More logs = more observable"	Unstructured noise makes incident handling slower, not faster. Three queryable events beat three hundred lines of prose.
"console.log is good enough for now"	Unstructured output can't be filtered, correlated, or alerted on. A structured logger costs five extra minutes once.
"We'll just look at the dashboard when something goes wrong"	A dashboard with no defined question shows you everything except the answer. Start from the on-call question.
"Alert on everything unimportant and tune it later"	A noisy pager teaches people to ignore it. Tuning never happens; the real page that gets missed does.
"Making User ID a metric label is more convenient for debugging"	It will also bring your metrics backend crashing down. High-cardinality lookups belong in logs and traces.
"We only have two services, tracing is overkill"	Two services already means cross-service latency problems that logs cannot answer. Auto-instrumentation makes this cost negligible.

Red flags

• A feature PR with retries, queues, or external calls, yet no new telemetry added
• Log lines assembled with string interpolation instead of structured fields
• No correlation/request ID—every log line is an orphan
• A single log stream written to by the scheduler, webhooks, and manual runs at the same time, yet no field can say who produced the line
• Metrics labeled with user ID, raw URL, or error message text (cardinality bomb)
• Latency tracked only by averages, with no percentiles
• Alerts that fire every day, get acknowledged, and lead to no action
• While the user-facing error rate goes unmonitored, alerts on causes (CPU, memory) page a human
• Secrets, tokens, or full request bodies appearing in logs
• Betting the evidence that a production feature is healthy on "It works on my machine"

Verification

After instrumenting a feature, confirm:

• The on-call questions for this feature have been written down, and every signal maps to one of them
• All log output is structured (JSON), has stable event names, and every line carries a correlation ID
• Every log sink written to by multiple entry points carries an entry point field: set where the run starts and propagated along with the correlation ID, rather than inferred downstream
• No line of logs contains secrets, tokens, or unmasked PII (spot-check the actual output)
• Every new endpoint and every external dependency has RED metrics, with a bounded label set
• Latency is a histogram; p95/p99 are queryable
• A single request can be traced end to end in the tracing UI, with no broken spans
• Every new alert is symptom-based, has a runbook link, and has been test-fired once
• A failure artificially induced in staging can be localized from telemetry alone, without reading source code

For a quick overview version of this checklist (including a pre-release instrumentation gate), see ../../references/observability-checklist.md。