logging-backend

Logging Backend - Wide Events

Implement Wide Events (also known as Canonical Log Lines): one structured event per request that contains ALL the relevant context in a single record.

Core Concept

A Wide Event is a single JSON log that captures the complete lifecycle of a request. Instead of multiple scattered logs ("user logged in", "fetched data", "returned response") emit ONE event at the end with all the aggregated information.

Benefits:

Debug with ONE query instead of correlating multiple logs
Analysis with simple SQL over structured fields
No loss of context between logs
Effective tail sampling (only keep interesting events)

Comparison: Traditional log vs Wide Event:

// Log tradicional - multiples lineas dispersas
[10:30:00] INFO: User 789 logged in
[10:30:01] INFO: Fetching orders for user 789
[10:30:02] DEBUG: Query executed in 45ms
[10:30:02] INFO: Found 3 orders
[10:30:03] ERROR: Failed to send email notification

// Wide Event - todo en un solo registro
{
  "timestamp": "2024-01-15T10:30:03.000Z",
  "user": { "id": "789" },
  "action": "fetch_orders",
  "result": { "count": 3 },
  "infra": { "db_time_ms": 45 },
  "error": { "type": "email_failed", "message": "SMTP timeout" }
}

Structure of a Wide Event

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "info",
  "service": "api-gateway",
  "trace_id": "abc123",

  "request": { "method": "POST", "path": "/api/orders", "duration_ms": 245 },
  "user": { "id": "user_789", "plan": "premium" },
  "business": { "order_id": "ord_456", "total": 150.00, "items_count": 3 },
  "infra": { "db_queries": 4, "db_time_ms": 120, "cache_hits": 2 },
  "error": null
}

See references/wide-event-schema.md for the complete schema with all recommended fields.

Implementation Workflow

Phase 1: Base Middleware

Create middleware that initializes the wide event at the start of the request
Attach the event to the context (req.wideEvent, AsyncLocalStorage, etc.)
Emit the log when the request finishes

See references/implementation.md for the middleware code.

Phase 2: Enrichment

In each handler/service, add fields to the wide event
Use typed helpers: enrichUser(), enrichBusiness(), enrichError()
Accumulate metrics: queries, cache hits, external calls

Phase 3: Tail Sampling

Implement a sampling function that decides what to log
Always log: errors, high latency, specific users
Sample a percentage of normal successful requests

See references/implementation.md for the tail sampling function.

Phase 4: Colorization (Development)

Configure colors by level (ERROR=red, WARN=yellow)
Configure colors by context (Request=blue, User=green)
Format readable output in the terminal

See references/colorization.md for color configuration.

Quick Checklist

Middleware that creates a wide event per request
Context accessible throughout the chain (AsyncLocalStorage / req)
Request fields: method, path, status, duration_ms
User fields: id, plan, roles (if applicable)
Business fields: domain entities
Infra fields: db_queries, cache_hits, external_calls
Error fields: code, message, stack (if there is an error)
Tail sampling configured
JSON format in prod, colored in dev

Anti-patterns

See references/anti-patterns.md for common mistakes:

Confusing structured logging with wide events
Relying only on OpenTelemetry
Logging multiple times per request
Not including business context

References

references/wide-event-schema.md - Complete JSON schema with all fields
references/implementation.md - Middleware and helpers code
references/anti-patterns.md - What NOT to do
references/colorization.md - Color configuration for dev

Troubleshooting

The wide event does not capture user data

Symptom: user: null in all logs

Common cause: The auth middleware runs after the wide event middleware.

Solution:

// Orden incorrecto
app.use(wideEventMiddleware('my-service'));
app.use(authMiddleware); // user no disponible cuando se crea el evento

// Orden correcto
app.use(authMiddleware);
app.use(wideEventMiddleware('my-service')); // user ya esta en req

Logs come out empty in infra.db

Symptom: db: { queries_count: 0, total_time_ms: 0 }

Common cause: The tracking helpers are not being used.

Solution: Wrap the queries:

const query = async (sql: string) => {
  const start = Date.now();
  const result = await db.query(sql);
  trackDbQuery(Date.now() - start);
  return result;
};

Logs too large in production

Symptom: High storage/ingestion costs

Solution: Implement aggressive tail sampling:

const shouldLog = (event: WideEvent): boolean => {
  if (event.level === 'error') return true;
  if (event.request.duration_ms > 1000) return true;
  return Math.random() < 0.01; // Solo 1% del resto
};

I can't correlate logs between services

Symptom: Each service has its own trace_id

Cause: The header is not being propagated.

Solution: See references/implementation.md section "Cross-Service Propagation".

About this skill