Division of labor with alert-triage :alert-triage is a single-track flow, it only consumes Grafana alerts, and only decides benign vs real./rca is the orchestration layer: the input additionally includes Sentry and CI, four tracks investigate simultaneously, and during synthesis it enforces contradiction adjudication and drift classification.alert-triage 's KB priors comparison tableand grafana MCP tool gotcha are reused directly (~/.claude/skills/alert-triage/SKILL.md Step 2, Tools section), not copied again here.
Hard rules (apply at every step)
- All four tracks are read-only. Investigation agents use
readonly-verifier(no Write/Edit),model: sonnet. Prohibited:kubectl apply/patch/delete/rollout、terraform apply/import/state rm、git push, any POST/PUT to Sentry/Grafana.terraform planalways carries-lock=false -refresh=false; when a refresh is needed, use-refresh-only -lock=falseinstead, and only read the output, do not apply. - subagents have no MCP（learned
subagents-do-not-inherit-mcp-servers-hand-them-an-http-helper). The dispatch prompt must not writemcp__*, the data channel is alwaysscripts/obs_http.py：
When dispatching, also instruct: do not cat this helper, do not print the header, do not printH=~/.claude/skills/rca/scripts/obs_http.py python3 $H sentry get 'issues/<id>/' # issue 詳情 python3 $H sentry get 'issues/<id>/events/?full=true' # 事件（含 release／tags／stack） python3 $H grafana get /api/datasources # 找 datasource uid python3 $H grafana prom <uid> '<promql>' --start <ISO> --end <ISO> --step 60 python3 $H grafana loki <uid> '<logql>' --start <ISO> --end <ISO> --limit 200~/.claude.json。 - numbers, no adjectives. Every finding must include "command + key output + time window (UTC)"; anything that says "a large amount", "occasional", "recently" without numbers is treated by SYNTHESIS as having no evidence.
- Anti-pattern guardrails(both are pitfalls actually stepped on):
- Do not use a literal resource name grep to determine whether it is managed by IaC. Resource names are often assembled from
"${var.env}-${local.name}"、for_each, module outputs. The order of determination is:terraform state list | grep(authoritative source) →terraform state show <addr>→ reverse-lookup module/variables/*.tfvarsto find out how the name is assembled. Only when "it cannot be found in state,andall naming paths have been ruled out" can it be called unmanaged. Previously a literal-name grep was used, and a certain bucket was misjudged as not managed by terraform. - Do not report a secret without first reading the value. When you see a field that looks like a key/token, first distinguish whether it is a placeholder (
changeme、<TOKEN>、${...}、xxx, ESO/SOPS reference, empty string) or a real value. Reading the value is only to determine the type; in the report write only "real value/placeholder/reference" and the length,do not output the value itself. If the value has not been read, writeunverified。
- Do not use a literal resource name grep to determine whether it is managed by IaC. Resource names are often assembled from
- ownership: when encountering
smb-*／payment-*'s helm values, secrets, backend EC2, someone else's PR status, only investigate, only propose, do not fix on their behalf (rulesrepo-ownership.md）。 - timezone: always convert to UTC to align the timeline; put Taiwan time (+08:00) in parentheses.
Step 0 — Normalize the alert (done in the main conversation, MCP allowed)
Determine the input type, extract envelope, write to .verification/<YYYY-MM-DD>/rca-<slug>/00-envelope.md：
| Input | How to extract | envelope required |
|---|---|---|
| Sentry URL／shortId | mcp__sentry__get_sentry_issue＋get_sentry_event_details(latest and earliest) |
title, culprit, first/lastSeen, count, firstRelease/lastRelease, environment, top 5 frames of the stack,Other issues with the same title（list_sentry_issues query is:unresolved "<title>"; when Sentry splits groups by fingerprint, list them all) |
| Grafana alert URL | grafana_api_request /api/prometheus/grafana/api/v1/alerts＋alerting_manage_rules get |
alertname, labels, original rule query text, activeAt, datasource uid |
| CI run URL | gh run view <id> --repo <r> --log-failed | tail -200、gh run view --json headSha,headBranch,createdAt,event |
repo, workflow, branch, headSha, failed job/step, error summary,sha of the previous green run（gh run list --workflow <w> --branch <b> --status success -L 1） |
The envelope must additionally define:alert time window(first seen −2h to last seen, max 7d),code paths involved(stack frame, culprit, failed test files),repos and infra components involved. These three are the shared scope for all four routes; if the scope cannot be determined, first AskUserQuestion, don't let the four routes each guess on their own.
Step 1 — Parallel dispatch (send four Agent calls in the same message)
Expecting ≥2 agents in parallel; per global rules, first use AskUserQuestion to ask whether to enable the formation (with token estimate: 4 Sonnet routes at about 60–120k each, 1 SYNTHESIS Opus route at about 100k). If the user has already agreed in the same request, don't ask again.
Common skeleton for each route's prompt: full envelope text, hard rules 1–6,H= helper usage, output format (see below),"give the conclusion first, ≤5 lines". Each route's report format is fixed:
## <LANE> 結論（≤5 行）
## 發現
| # | 發現 | 時間 (UTC) | 證據指令 | 關鍵輸出（原文節錄） | 信心 H/M/L |
## 沒查到／查不了的
## 我認為其他路會看到什麼（供 SYNTHESIS 對賭）
The last paragraph is deliberately designed: each route is required to predict the other routes' results, so that SYNTHESIS has something to cross-check against and find contradictions.
| Lane | Scope | Must do | Must not do |
|---|---|---|---|
| OBSERVABILITY | metrics/logs/traces/Sentry events around the alert time window | time distribution of event counts (hourly buckets), affected release/browser/URL distribution (Sentry tags), error rate/latency/restart count for the same period; every number must include the query |
infer code root cause |
| HISTORY | repos involved, commits in the last 30 days that touched the paths involved | git fetch after git log --since=30.days --format='%h %ad %an %s' --date=iso -- <paths>; map Sentry release sha/CI green-red sha to commit ranges (git log <good>..<bad>); dependency changes (package.json/lockfile diff); list candidate commits and explain why they are suspicious |
modify code, checkout another branch (use git show／git log, do not switch worktree state) |
| KNOWLEDGE | ~/Documents/knowledge-base |
use BM25 to run bm25_retrieve.py "<症狀>"、--index wiki/rules/INDEX.md、wiki/decisions/INDEX.md；grep -rn '\bAD-[0-9]\+' to find related AD-numbered decisions,quote the original text verbatim(file:line); past similar reports/; also specifically look for notes saying "we already decided this/by design/intentional" |
rewrite or summarize the AD original text |
| INFRA | live state of cluster/CDN/IaC | kubectl config current-context first confirm context; nodes/events (spot interruption, Karpenter disruption), PDB and orphaned resources whose selector does not match, ingress/CF cache rule;terraform plan -lock=false -refresh=false or state list compare drift; S3 lifecycle rules;the two items in guardrail 4 must be followed |
any write action; backend-owned resources are only to be looked at, not evaluated |
When INFRA determines "unrelated to this alert," it must write out the basis for exclusion (for example: node event count within the time window is 0, with command attached); it cannot simply leave it blank.
Every route's prompt must explain:Before replying, first use a Bash heredoc to write the complete report verbatim to .verification/<date>/rca-<slug>/{1-obs,2-history,3-knowledge,4-infra}.md. Do not rely on the main conversation to transfer it, because a teammate's idle notification will truncate long reports (this happened on the first run on 2026-09-22). The main conversation only verifies that the file exists and that the line count is reasonable.
Step 2 — SYNTHESIS (dispatch 1 Opus agent, fresh context, given only the five file paths)
What is handed to SYNTHESIS is the file paths, not a summary from the main conversation, to avoid passing down the main conversation's preconceptions. It must produce, in order:
- Merged timeline: a table, with columns being
時間(UTC) | 事件 | 來源 lane | 證據. Deploys, commits, merges, first seen of alerts, numeric turning points, and infra events must all be included. - Contradiction list and adjudication: list every set of "A lane says X, B lane says Y" (including cases where one lane's prediction does not match another lane's measurement).Adjudication may rely only on new evidence: SYNTHESIS itself runs one read-only command (helper/git/kubectl/grep) to resolve it, and writes the command and output into it.Forbiddento adjudicate using "A's confidence is higher" or "A's description is more specific." For anything new evidence also cannot resolve, mark
UNRESOLVED, and write what evidence would be needed to resolve it. - Root cause: state the root cause in one sentence, with confidence (HIGH/MEDIUM/LOW + reason), and writethe single piece of evidence that could overturn it(falsifier), in the format "if the result of
<指令>is<X>, this root cause does not hold." If you cannot write a falsifier, confidence can be at most LOW. - Classification(choose exactly one, sub-signals may be split out):
SELF-INFLICTED → CODIFY: someone manually changed live state (console,kubectl edit, CF dashboard), and IaC/repo has not caught up yet. The remedy isto write the live state back into IaC or the repo, not to apply the IaC back over it. First confirm that this manual change was intentional (KNOWLEDGE's AD/decision notes, commit message, user).DRIFT → REVERT: live state has drifted, and this drift is wrong, so live must be brought back to the IaC definition.REAL DEFECT → FIX: the code or configuration itself is defective.EXTERNAL／BENIGN → RECORD: third party, the user's environment (browser extensions, etc.), or by design; only record it, do not open a PR. This step must explain why it is not the other three categories.Pay special attention: when you see drift, the default question is "is this drift the correct state that should be written into terraform," not "should it be applied back." We have been burned before: drift that should have been committed into terraform was directly applied as if it were the normal state.
- Fix PR draft + rollback checklist: write it as a draft file, do not open a PR (opening a PR requires HITL G1). The draft includes repo, base branch (
vocus-web-uiusehotfix), the diff of changes or specific file:line numbers, a Traditional Chinese PR body, and acceptance method (must be observable: which query's number drops to what). The rollback checklist includes: trigger condition (which number rises back to what), the command for each step, the expected output of each step, whether data/state is reversible, and whom to notify.
SYNTHESIS's output is written to the KB's reports/<YYYY-MM-DD>-rca-<slug>.md（frontmatter type: incident-rca, with tags carrying rca), and attached to reports/INDEX.md. The PR draft is placed in the same directory's .verification/<date>/rca-<slug>/pr-draft.md。
Step 3 — Acceptance and HITL
- Dispatch
readonly-verifier(fresh context) Spot-check: pick 3 timeline entries, all contradiction rulings, and actually run the falsifier commands once each, marking PASS/FAIL item by item, with evidence stored.verification/. If there is any FAIL, go back to Step 2 and fix it. Only with zero FAIL can the report's status be written asverified。 - Before G1 opens the PR、G3 touches resources owned by the backend、G4 root-cause confidence LOW: use AskUserQuestion to stop and wait for the user to decide. When the classification is
SELF-INFLICTED → CODIFY, additionally ask the user, "Was this manual change something you did intentionally?", unless KNOWLEDGE has already found an explicit written decision. - KB gate: before recycling the agent, write this lesson to
_pending/session-<date>-rca-<slug>.md; write reusable practices towiki/learned/, and attach them to INDEX.
Format for delivery to the user
Lead with the conclusion (≤5 lines): root cause, confidence, falsifier, classification, and what the user needs to decide next. Attach the report path and PR draft path. Give details only after the user says "expand".