Use this skill for read-only Datadog observability investigations.
Reference loading
Load references conditionally based on the request:
| Need | Read |
|---|---|
| Any Datadog operation | references/api-surface.md |
| Log search, metric query, trace lookup, incidents | references/common-use-cases.md, references/query-syntax.md |
| Auth failures, permission errors, or tool failures | references/troubleshooting-workarounds.md |
Workflow
- Resolve the operation and target:
-
Determine whether the request is a log search, metric query, trace/span inspection, monitor lookup, incident lookup, dashboard/notebook lookup, service/host listing, or service-dependency map.
-
Prefer explicit env, service, host, monitor/incident IDs, trace IDs, or Datadog URLs when the user provides them.
-
When the user did not specify a scope, treat
datadog.envanddatadog.serviceconversation config as optional defaults. Explicit user input always wins over config. -
Only set or change
datadog.envanddatadog.servicewhen the user explicitly asks to store a default for this conversation or channel. -
If the request refers to an earlier telemetry item indirectly, inspect the current thread for the existing ID or URL before asking the user to restate it.
-
Ask one concise follow-up only when a search is genuinely under-specified, for example when the user asks about "errors" with no env, service, or time window hint and the thread has no prior context.
-
If an active repository context exists (cloned repo or configured
github.repo), check the repo root forTELEMETRY.mdbefore forming queries. When present, use its query recipes, service/env mappings, and investigation pivots as repo-specific guidance. Explicit user targets, IDs, URLs, and conversation config still win. If absent, continue normally.
- Use Pup:
- Run Datadog commands with
pup --read-only --agent .... The plugin also sets read-only/agent env vars, but include the flags so command transcripts show the intended mode. - If you are unsure about a command or flag, inspect Pup's schema with
pup --read-only --agent agent schema --compactor the relevantpup --read-only --agent <group> --helpoutput before guessing. - Start narrow: pick the single most direct command for the request before broader search.
- Known incident ID:
pup --read-only --agent incidents get <incident_id> - Known monitor ID:
pup --read-only --agent monitors get <monitor_id> - Known notebook ID:
pup --read-only --agent notebooks get <notebook_id> - Known metric name:
pup --read-only --agent metrics query --query="avg:<metric>{...}" --from="15m" --to="now"; usemetrics metadata getormetrics tags listwhen the user wants available tags or dimensions.
- Known incident ID:
- For exploratory questions, prefer one focused Pup search/list/aggregate command, then one follow-up fetch if needed.
- For "current error rate / log volume / top offenders" questions, prefer
pup logs aggregateover pulling raw log pages back throughpup logs search. - For service-topology questions ("what calls checkout?", "what does the payment API depend on?"), prefer
pup apm dependencies listorpup apm flow-mapover stitching spans together manually. - Use
pup monitors searchorpup monitors listfor "is this alerting?" andpup incidents list/pup incidents getfor incident context. - Use RUM commands only when the user asks about real-user / browser telemetry, not for backend issues.
- Bound every query:
- Always constrain time windows. Default to the last 15 minutes for "right now" questions and the last 24 hours for retrospective questions; otherwise use the window the user named.
- Always include
env:whendatadog.envis set or the user named an env. - Always include
service:when the user named a service ordatadog.serviceis set and the command is service-scoped. - Cap result size. Prefer the default or small page sizes; do not page through thousands of logs when an aggregate command answers the question.
- Report the result:
- Return the concrete answer first (counts, status, incident severity, trace timing, top offenders), then a short evidence block.
- Include Datadog deep links when Pup returns them or when you can construct a stable app link from an ID. Do not fabricate links from incomplete identifiers.
- Preserve interesting spans, log lines, or metric values inline only when they are evidence for the answer. Do not dump raw command output.
- Keep routine tool chatter silent. Do not narrate every Pup search or fetch step.
Guardrails
- Read-only only in this skill. Do not create, edit, mute, delete, import, submit, or resolve monitors, incidents, notebooks, dashboards, SLOs, metrics, API keys, RUM resources, or other Datadog objects.
- Log, RUM, APM, and incident payloads can contain PII or sensitive customer data. Quote only the minimum needed to answer the question. Do not paste full raw log bodies or span payloads when a summary plus a deep link is enough.
- If Pup returns
403,permission denied, or similar, stop and tell the user the Datadog credentials could not access the requested resource. Do not guess at missing RBAC scopes. - If Datadog responds with
429 Too Many Requests, wait briefly and retry the same query once. If it still fails, report the throttle and stop. - For large traces or span responses that are incomplete, report that fact; do not pretend the shown spans are complete.
- Do not use this skill for Sentry issues, Linear/GitHub ticketing, or source-code investigation. Hand those off to the matching skill.