Workflow
- Scope the request. Identify the signals, deployment target, Collector/SDK version, backend protocol, expected volume, outage tolerance, and trust boundaries from the supplied files or context. Ask only for missing details that change the design; state assumptions for a draft.
- Load relevant references. Use the trigger table below. For version-sensitive settings, check compatibility.md and the upstream documentation for the installed release.
- Choose the work mode. For existing configuration, apply the review checks below before editing. For a new Collector pipeline, load production-baseline.md and the platform setup guide; adapt their examples to the user's environment. For instrumentation, load the matching SDK/agent reference. For missing or dropped telemetry, follow validation.md from source through Collector to backend.
- Validate and correct. For Collector changes, run
otelcol validate --config <path>with the target distribution/version; render Helm values before validating the resulting config. For SDK changes, exercise a representative request and inspect emitted telemetry. Fix reported failures and repeat the affected check; if blocked, report the exact failure and remaining verification. - Deliver the result. Include the findings or changed config/code, reasons for consequential choices, and checks performed with their results. Separate configuration validity from observed end-to-end delivery; do not claim live recovery from parsing or a health endpoint alone.
Design Guardrails
- Collector memory: put
memory_limiterfirst in every pipeline to prevent OOM crashes; size it below the container/host limit with runtime and buffer headroom. Prefer backpressure or controlled drops over crashing the Collector. - Metric cardinality: reject unbounded
user_id,request_id,session.id, ortrace_iddimensions and explain time-series growth. Suggest privacy-safe traces/logs plus bounded metric labels; use the Rule of 100 in instrumentation.md as the review heuristic. - Transport: default to OTLP gRPC (4317); use OTLP HTTP (4318) when the client, proxy, browser, or backend requires it.
- Security: redact PII; use TLS and authentication across trust boundaries, with mTLS where mutual peer identity is required. Keep health/debug endpoints private; never expose pprof or zpages publicly.
- Conventions and stability: prefer OpenTelemetry Semantic Conventions; verify component stability for the selected release and warn about non-stable production dependencies.
- Routing: use stable keys (
traceIDfor tail sampling,tenant_idorclusterfor tenant/shard routing); normalize non-string attributes first. - Kubernetes tail sampling: use a Gateway Deployment behind
load_balancingwithrouting_key: traceIDand a Headless Service (clusterIP: None). For keep-errors-plus-10%, include error and probabilistic policies. Explicitly warn thattail_samplingis Beta in Collector 0.160; recheck stability for other releases.
AI Agent Instrumentation
For coding-agent requests, load ai-agents.md and apply these signal-specific constraints:
- Claude Code telemetry: include
CLAUDE_CODE_ENABLE_TELEMETRY=1and managed-settings persistence; choose metric temporality for the backend (default delta, cumulative when required); traces are beta, while metrics and logs/events remain the broadly documented signals. Keep prompt/tool-content capture disabled unless PII controls are explicit, and avoidsession.idas a metric dimension. - Cursor telemetry: distinguish Enterprise native metrics/logs from optional hook-derived traces; load the Cursor section before configuring export or combining the sources.
- AI agent tool-call tracing: for agents using current
gen_ai.*conventions, setgen_ai.operation.name: execute_tool, preservegen_ai.tool.name, and useexecute_tool {gen_ai.tool.name}for manually generated tool spans; keep the registered tool name in the attribute as well. These GenAI conventions are Development; preserve vendor-native spans. - GenAI provider vs agent identity: preserve
gen_ai.provider.namefor the model/provider (for example,openaiorgcp.gen_ai); useservice.nameor a natively emitted agent attribute for the coding-agent identity rather than writing the agent name into the provider field.
Existing Configuration Review Mode
Check these interactions, beyond configuration syntax:
- Memory vs pod limit — compare the limiter's actual ceiling with the pod limit, not just the configured percentage.
- Stateful processing vs scaling —
tail_sampling,span_metrics, andservice_graphneed sticky routing above one replica. - Durability vs outage tolerance — disabled retries and no persistent queue mean data loss on backend failure.
- Deployment mode vs exposure —
hostPortfits DaemonSet/node-local patterns, not scaled gateway Deployments. - Rollout settings — review
replicaCount, HPAminReplicas, PodDisruptionBudget, and rolling updates together. - OTTL/filter correctness — keep attribute types consistent and prefer current semantic convention keys.
- Metric temporality and state —
deltatocumulative/cumulativetodeltaneed source/backend/restart checks. - Queue storage backend —
file_storageneeds local locking-safe storage; RWX, EFS, and NFS are unsafe defaults. - Generated metric identity — for log/span-to-metric connectors, verify replica count, export mode, actual temporality, whether the emitted producer identity survives backend mapping, and the aggregation query shape together.
Progressive Disclosure: Context Triggers
Load only the references matching the request:
| Trigger keywords | Load | Key topics |
|---|---|---|
| New Collector pipeline, baseline config | production-baseline.md | Complete OTLP config, required substitutions, storage and listener setup |
| Choose deployment platform | setup-index.md | Platform decision matrix |
| Deploy to Kubernetes, Helm values.yaml | setup-kubernetes.md | DaemonSet, Gateway and sidecar manifests |
| Deploy to ECS, Fargate | setup-ecs.md | Task definitions, IAM and secrets |
| Deploy to Docker, Compose | setup-docker.md | Container networking, resources and volumes |
| Deploy to VM, EC2, systemd, Windows service | setup-vm.md | Service lifecycle and host setup |
| Kubernetes, Helm, values.yaml, audit, review, DaemonSet, Sidecar, Gateway, Scaling, Load Balancing | architecture.md | DaemonSet vs Gateway vs Sidecar, Target Allocator, HPA, rollout consistency |
| Pipeline, Receiver, Processor, Exporter, Queue, Batch, Memory, Extensions, existing config | collector.md | Processor ordering, memory_limiter, file_storage, config audit heuristics, temporality/state audits, stability levels |
| Python, FastAPI, Starlette, asyncio, Python GenAI, Python SDK events | python-instrumentation.md | Initialization ownership, duplicate instrumentation, SDK 1.44 migration, streaming, GenAI packages |
| JavaScript, Node.js, OTel JS 3.0, tracing SDK migration | javascript-otel3-readiness.md | Runtime floor, removed packages, tracing SDK imports, custom instrumentation review |
| SDK, Instrumentation, Spans, Attributes, Semantic Conventions, Cardinality | instrumentation.md | Auto vs manual, SemConv, cardinality Rule of 100 |
| Sampling, Cost, Volume, Head Sampling, Tail Sampling, Probabilistic | sampling.md | Head/tail sampling, sticky sessions, sampling math |
| Security, PII, GDPR, Redaction, TLS, Authentication, Credentials | security.md | PII redaction, mTLS, RBAC, extension exposure risks |
| Monitor the collector, Health, Alerts, Self-monitoring, Collector metrics | monitoring.md | otelcol_* metrics, dashboards, alert rules |
| Lambda, Azure Functions, GCP Functions, Serverless, FaaS, Mobile, Browser | platforms.md | FaaS patterns, Lambda extension layer, client-side apps |
| OTTL, Transform, Transformation, Modify, Filter attributes, Parse, Extract | ottl.md | OTTL syntax, context types, built-in functions, error handling |
| Connector, span_metrics, service_graph, signal_to_metrics, log-to-metric, span-to-metric, routing connector, failover connector | connectors.md | R.E.D. metrics, service graph, routing, failover, stickiness, generated-metric producer identity |
| Claude Code, Codex, Gemini CLI, Antigravity, Copilot, Cursor, AI agent, coding agent, MCP | ai-agents.md | Agent OTel support matrix, unified collector config, GenAI SemConv |
| validate, dry-run, startup error, pipeline error, dropped data, queue full, recovery | validation.md | Config validation commands, live checks, symptom→cause→fix recovery guidance |
| playbook, production playbook, blog, developer observability, local OTel viewer, real world | playbooks.md | Production and developer patterns from OpenTelemetry and CNCF blogs |
| anti-pattern, common mistake, what to avoid, pitfall | anti-patterns.md | Full annotated anti-pattern catalogue: pipeline, metrics, Kubernetes, AI agents, OTTL |