observability-apm-expert

Observability & APM Expert

Implement comprehensive observability with distributed tracing, metrics, structured logging, and SLO-based alerting using OpenTelemetry and modern backends.

Decision Points

1. Sampling Strategy Selection

Error Rate Analysis:
├── Error rate < 0.1%
│   ├── Low cardinality service (< 10k spans/min) → 100% sampling
│   └── High cardinality service (> 10k spans/min) → Tail-based sampling
│       ├── Keep all error traces (100%)
│       ├── Keep slow traces > P95 latency (100%)
│       └── Sample successful traces (1-10%)
└── Error rate > 0.1%
    ├── Critical service → Keep all errors + 50% successful
    └── Non-critical service → Keep all errors + 10% successful

2. Backend Selection Strategy

If self-hosted tolerance = high AND cost sensitivity = high:
├── Use Grafana stack (Tempo + Mimir + Loki)
└── Export via OTLP to unified collector

If operational overhead tolerance = low OR compliance = strict:
├── Cloud vendors (Datadog, New Relic, Honeycomb)
└── Direct SDK exports + OTLP fallback

If hybrid requirements:
├── Critical services → SaaS backend
└── Development/staging → Self-hosted stack

3. Alert Configuration Logic

For each SLO:
├── Define error budget (e.g., 99.9% = 43.2min downtime/month)
├── Calculate burn rates:
│   ├── Fast burn (14.4x) over 1h → Critical alert (2min delay)
│   ├── Medium burn (6x) over 6h → Warning alert (15min delay)
│   └── Slow burn (3x) over 24h → Info alert (1h delay)
└── Link each alert to specific runbook action

Failure Modes

Schema Bloat

Symptom: Metrics cardinality > 10M series, query timeouts, high storage costs Detection: prometheus_tsdb_head_cardinality growing exponentially Fix: Add label cardinality limits, aggregate high-cardinality labels, use recording rules

Trace Orphaning

Symptom: Spans appearing disconnected, missing parent-child relationships Detection: Spans with same trace_id but no parent reference in service map Fix: Verify context propagation headers (traceparent/tracestate), check async context handling

Alert Fatigue Storm

Symptom: > 10 alerts per incident, team ignoring notifications Detection: Alert:incident ratio > 5:1, MTTA (time to acknowledge) > 30min Fix: Implement alert dependencies, use SLO burn rate instead of threshold alerts

Sampling Blind Spots

Symptom: Critical errors not captured in traces, debugging impossible Detection: Error logs present but corresponding traces missing Fix: Switch to tail-based sampling, increase error trace retention to 100%

Context Propagation Gaps

Symptom: Traces terminate at service boundaries, no cross-service correlation Detection: Spans from downstream services have different trace_id Fix: Verify HTTP headers propagation, add OTel middleware to all services

Worked Examples

Production Incident Investigation

Scenario: Customer reports 5-second checkout timeouts starting 2 hours ago

Step 1 - Triage with SLO dashboard:

P99 latency jumped from 200ms → 5000ms at 14:30 UTC
Error rate spiked from 0.1% → 2.3%
SLO burn rate: 46x (critical threshold)

Step 2 - Trace analysis:

-- Find slow traces in time window
{service_name="checkout-service"} |= "POST /checkout" 
| json | duration > 2s | trace_id

Expert insight: Filter by duration first, then sample traces - don't analyze all traces

Step 3 - Root cause drill-down:

Selected trace_id: abc123
├── checkout-service: 50ms (normal)
├── payment-service: 4.8s (🚨 anomaly)
│   ├── validate_card: 45ms
│   ├── fraud_check: 12ms  
│   └── database_query: 4.7s (🚨 root cause)
└── inventory-service: 100ms

Step 4 - Correlate with infrastructure:

Database span attributes show:
- db.statement: "SELECT * FROM transactions WHERE user_id = ?"
- db.connection.pool.idle: 0
- db.connection.pool.max: 10

Expert insight: Connection pool exhaustion - scale pool or optimize queries

Resolution: Increased connection pool from 10 → 50, added query timeout

Custom Instrumentation Setup

Scenario: Add business metrics for order processing pipeline

// 1. Initialize custom meter
import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('order-service', '1.0.0');

// 2. Define business metrics
const ordersTotal = meter.createCounter('orders_total', {
  description: 'Total orders processed',
  unit: '1'
});

const orderValue = meter.createHistogram('order_value_dollars', {
  description: 'Order value distribution',
  unit: 'USD'
});

// 3. Instrument business logic
async function processOrder(order: Order) {
  const span = trace.getActiveSpan();
  span?.setAttributes({
    'order.id': order.id,
    'order.user_id': order.userId,
    'order.value': order.totalValue
  });

  try {
    // Business logic here
    await validateOrder(order);
    await chargePayment(order);
    
    // Record success metrics
    ordersTotal.add(1, { 
      status: 'success',
      payment_method: order.paymentMethod 
    });
    orderValue.record(order.totalValue);
    
  } catch (error) {
    span?.recordException(error);
    span?.setStatus({ code: SpanStatusCode.ERROR });
    ordersTotal.add(1, { status: 'error' });
    throw error;
  }
}

Quality Gates

• All HTTP requests include traceparent header propagation
• Every log statement contains trace_id and span_id fields
• RED metrics (Rate/Errors/Duration) available for each service endpoint
• P99 latency SLO defined with < 2% error budget burn rate alerting
• Tail-based sampling retains 100% of error traces and slow traces
• Database queries instrumented with connection pool and query performance spans
• Custom business metrics tagged with bounded cardinality labels (< 1000 values)
• Alert runbooks linked and tested for each SLO violation scenario
• Trace sampling decision logged and queryable for debugging
• Cross-service dependency map auto-generated from span relationships

NOT-FOR Boundaries

Application Error Handling → Use relevant language skill (node-js-expert, python-expert, etc.) for try/catch, error boundaries, graceful degradation

Security Event Monitoring → Use security-expert skill for SIEM, threat detection, compliance logging

Log Storage Infrastructure → Use kubernetes-expert or cloud-expert for ELK stack deployment, log retention policies

Performance Testing → Use load-testing-expert for generating telemetry during performance validation

Cost Optimization → Use finops-expert for observability spend analysis and retention tuning