log-diagnosis

Log Diagnosis — CloudWatch Log Diagnosis

Query logs via CloudWatch Insights, match known error patterns, and output a structured diagnostic report.

Execution Mode

• Single service / one or two log groups → inline: Just follow the workflow below; the main agent can query serially, which is fastest.
• Multiple services / global scan → workflow fan-out: When the scope is multiple services or "all services", the main agent first completes Step 0's scope confirmation (this step must be inline, because it requires asking the user questions + parsing log groups from account.yaml), then hands the resolved scope to assets/multi-service-scan.workflow.js for parallel processing. See "Multi-Service Fan-Out" below for details.

Workflow

Step 0 — Confirm Scope (ASK FIRST)

MUST:

• Confirm the target service/project → locate from environments/<account>/account.yaml 's service_catalog
• Confirm the time range → defaults to the past 1 hour
• Confirm the region → defaults to ap-northeast-1
• If the user says "all services" or "global", scan each service in the service_catalog one by one

SHOULD:

• If the user describes specific symptoms (e.g., "API 500 errors"), select a more precise query template in Step 2
• If the scope is multiple services or "all services": After completing this step's scope confirmation (service list, time window, region, log group for each service), do not scan them inline one by one; instead switch to the workflow in the "Multi-Service Fan-Out" section. The logic for Steps 1–4 is executed by subagents within the workflow, and the main agent is only responsible for rendering the final report.

Step 1 — Locate the Log Group

MUST:

• Read references/log-source-map.md to get the service → log group naming convention
• Use aws logs describe-log-groups --log-group-name-prefix to confirm the log group actually exists
• Check storedBytes > 0 (if the group exists but has no data, prompt the user)

SHOULD:

• If account.yaml has a log_sources section, prefer the mappings in it
• For unknown services, follow log-source-map.md 's auto-discovery process

Note: Do not construct log group names by guessing. Must confirm with describe-log-groups.

Step 2 — Execute the Query

MUST:

• From references/query-templates.md , select a suitable query template
• Use scripts/query-logs.sh to execute the query (encapsulates the async poll process)
• Do not write Insights queries yourself—use the verified queries in the template library
• If the template library has no exactly matching scenario, modify the closest template and record it in Gotchas

Script usage:

# 单个 log group
scripts/query-logs.sh "/aws/lambda/my-func" \
    'filter @message like /(?i)(error|exception|fatal)/ | fields @timestamp, @message | sort @timestamp desc' \
    1 ap-northeast-1

# 多个 log group
scripts/query-logs.sh --multi-groups "/aws/lambda/func-a,/aws/lambda/func-b" \
    'filter @message like /(?i)error/' \
    1 ap-northeast-1 --limit 50

SHOULD:

• First use the generic error scan (Template 1a) to get the overall picture, then drill down with service-specific templates
• For queries with a time range > 24h, first confirm the pattern with a short window (1h) before expanding

Step 3 — Analyze Results

MUST:

• Read references/error-patterns.yaml, and match the query results against known patterns
• Classify by severity: CRITICAL > HIGH > MEDIUM > LOW
• Sort by frequency and find top errors
• Check the time distribution—whether it is a burst (concentrated at a certain point in time) or persistent (evenly distributed)

SHOULD:

• For errors that do not match known patterns, use aws-knowledge MCP to search relevant documentation
• Check whether they are related—whether multiple errors have a causal relationship (e.g., timeout causing 5xx)

Step 4 — Output the Report

MUST:

• Use assets/diagnosis-template.md Format and output the diagnostic report
• Includes: error summary, top errors (with raw log snippets), timeline, affected resources, recommendations

SHOULD:

• Provide next-step recommendations (which service needs to be drilled into, whether the aws-troubleshooting skill needs to be triggered)

Multi-service fan-out (Workflow engine, opt-in)

When the scope is multi-service/global, use assets/multi-service-scan.workflow.js to fan out each service's diagnosis to an independent subagent and run them in parallel.Why use workflow instead of an inline loop: diagnosing each service is not simply running CLI commands; it requires reading logs, matching patterns, and judging time distribution—this falls under "each unit requires LLM reasoning", and the services are independent of each other and Insights queries are slow, so it is suitable for pipeline parallelization; pure metric aggregation is the kind of work that should be left to bash.

Trigger constraints: Workflow is opt-in; the user needs to explicitly include the "workflow" keyword, or the main agent proactively proposes it under a multi-service scope and calls it after user consent. Do not automatically spawn it every time someone says "check the logs".

Prerequisites for invocation (the main agent must do first)：

1. Complete the scope confirmation in Step 0 (clarify the service, time window, region).
2. From environments/<account>/account.yaml the log_sources , parse out each service's log group name (the workflow will use describe-log-groups again to verify existence, but do not let it construct names out of thin air).
3. Pass the parsing result as args .

args contract：

{
  "account_id": "123456789012",
  "region": "ap-northeast-1",
  "hours_back": 1,
  "skill_dir": "/path/to/sample-aws-ops-skills-for-agents/log-diagnosis",
  "services": [
    { "name": "payments-api", "log_groups": ["/aws/containerinsights/payments-eks/application", "/aws/eks/payments-eks/cluster"] },
    { "name": "orders-service", "log_groups": ["/aws/lambda/orders-service-*"] }
  ]
}

log_groups can be omitted—when omitted, the scan subagent will follow log-source-map.md to auto-discover and describe-log-groups verify.

Two-phase pipeline：

• Scan(one subagent per service): run query-logs.sh → match against the error-patterns.yaml regex → classify + judge time distribution → return structured findings. Mandatory: every finding must be traceable to the query output,sample_message is the raw log line, no fabrication allowed.
• Enrich(only for errors that did not match a pattern): query the aws-knowledge MCP for possible causes + documentation sources; if the MCP cannot find it, state so honestly and mark low confidence, do not fabricate.

Return value：{ scope, services: [<scan 结果，可能含 enrichment>] }. After the main agent receives it, use assets/diagnosis-template.md to render the final report—report rendering stays in the main loop, because it needs to apply templates to make human-readable judgments and routing recommendations (DEGRADED → aws-troubleshooting, etc.).

Guardrail alignment: the workflow's "conclusions must be traceable to data sources" and "prohibited from reporting AWS numbers from memory" are consistent with the CLAUDE.md reporting discipline; the anti-hallucination rules in the Enrich phase are in the same lineage as aws-troubleshooting.

Query reference

• CloudWatch Insights query templates:references/query-templates.md
• Known error pattern library:references/error-patterns.yaml
• Service → Log Group mapping:references/log-source-map.md

Do not write query commands yourself—use the validated templates in the reference files.

Automation Boundary

auto-safe

• aws logs describe-log-groups(discover log groups)
• aws logs start-query + get-query-results(CloudWatch Insights query)
• aws cloudtrail lookup-events(audit log query)
• Run scripts/query-logs.sh
• to generate the diagnostic report

human-required

• Log deletion / cleanup (delete-log-group, delete-log-stream）
• Retention changes (put-retention-policy）
• Subscription filter modifications (put-subscription-filter）
• Any write operation

Gotchas

• CloudWatch Insights queries have a 15-minute timeout limit. A large log group + a long time range may trigger a timeout → shorten the time window and retry
• like /error/ is case-sensitive, and you must use /(?i)error/ for case-insensitive matching
• A log group not existing does not mean the service does not exist—many AWS services do not enable logging by default (RDS slow queries, API Gateway, EKS control plane)
• The log group name for EKS container logs depends on the logging solution (Fluent Bit vs CloudWatch Agent); do not hardcode it
• query-logs.sh of --limit The parameter is appended to the end of the query; if the original query already has | limit, it will be overwritten
• stats queries (such as stats count(*) by bin(5m)) are not subject to the limit 1000, but fields queries are subject to this limit