eng-observability

Observability and Debugging Discipline

Intent

• Make it trivial to answer “what is happening” and “why” without attaching a debugger in production.
• Ensure logs, metrics, events, and traces capture user intent, environment, and failure context while protecting sensitive data.

Guiding Principles

1. Prefer structured logs + correlation IDs over ad-hoc strings.
2. Emit signals at every boundary (client, API, worker, contract invocation).
3. Include context (user/session/network/chain) necessary to reproduce issues.
4. Keep signal cost reasonable—throttle chatty paths, sample intelligently.
5. Build fast local debugging loops (trace replay, state inspectors, dev wallets).

Workflow

1. Identify critical paths affected and define success/error signals per path.
2. Add/extend tracing spans or log blocks with consistent field names.
3. Validate observability locally by simulating successes, errors, and timeouts; ensure signals reach the sink (console, APM, analytics, chain explorer).
4. Document dashboards, queries, or CLI commands useful for post-deploy verification.
5. For on-chain logic, emit events with canonical schema so downstream indexers can consume them.

Verification

• Run the code with verbose logging/tracing enabled; inspect outputs for clarity and privacy.
• Confirm metrics/counters appear where expected (APM, telemetry pipeline, analytics, chain explorer).
• Dry-run incident response: can you locate a test failure or simulated outage using only emitted signals? If not, iterate.