byted-volcengine-rootcause-analyzer

Alert Root Cause Localization and Merging Skill

Capability Positioning

This Skill is anorchestration layer for failure scenarios. It does not treat an alert as an isolated event, but instead assembles alerts, resource topology, anomalies in the same time window, and runtime evidence into a complete failure context.

Its responsibilities are:

• Extract instances, domains, IPs, product types, metrics, and time windows from alert text, phenomenon descriptions, or on-call conversations
• Automatically expand the upstream and downstream topology around the target resource to form the current failure context
• Check whether other resources in the same topology have related anomalies
• Distinguish root alerts, derived alerts, and noise alerts to avoid handling a group of related alerts separately
• Aggregate topology facts, runtime status, monitoring evidence, and API data completion results to provide a ranking of root cause candidates
• Output impact scope, priority handling nodes, recommended remediation actions, and a structured summary that can be escalated for handover

The goal of this Skill is not to "explain a certain alert", but toconverge scattered anomalies into a single business link issue, approaching the true source of the failure as quickly as possible.

Current Boundaries

The current version focuses on covering the following capabilities:

• Alert parsing
• Resource identification
• Topology expansion
• Same-topology anomaly merging
• Root cause candidate ranking
• Structured failure summary output

The following capabilities can serve as future extension inputs, butthey are not currently mandatory steps, and their availability cannot be assumed：

• Recent changes
• Historical failure cases
• Knowledge base experience
• Authorized execution of remediation plans and verification of recovery

One-sentence distinction:

• byted-volcengine-rootcause-analyzer: responsible for stringing "alerts + topology + anomaly merging + evidence" into a diagnostic conclusion
• It is not a unified alert platform, not the resource management center itself, and not an automated execution platform

Dependent Skills and Relationships

This Skill depends on the following four existing Skills:

1. byted-volcengine-topology-builder

Responsibilities:

• Collect Volcengine account asset snapshots
• Build topology.json、topology.md, graph files, and other basic topology data

Boundaries:

• Only responsible for "building out resource relationships"
• Not responsible for interpreting alerts, nor for providing root cause conclusions

When it is suitable to call:

• There is currently no topology covering the target resource
• The existing topology is too old or missing key nodes
• The link needs to be completed from resources such as instances, listeners, EIPs, VPCs, and security groups

2. byted-volcengine-topology-analyzer

Responsibilities:

• Locate the target node based on the existing topology
• Expand direct upstream and downstream, entry links, and affected resources
• Provide static impact surface analysis

Boundaries:

• Only consumes static topology, which is not equal to real-time health status
• Not responsible for directly querying monitoring time series or API details

When it is suitable to call:

• The primary alarm resource or candidate resource has been identified
• It is necessary to determine "whom this resource affects and by whom it is depended upon in the entire chain"

3. byted-volcengine-cloudmonitor

Responsibilities:

• Query the monitoring time series of the target resource
• Verify whether the anomaly persists, amplifies, or propagates synchronously

Boundaries:

• Only answer "how the metrics change"
• Cannot independently replace topology attribution

When it is suitable to call:

• Need to confirm whether the alarm is still ongoing
• Need to verify whether multiple resources in the same topology are anomalous synchronously within the same time window
• Need to distinguish transient jitter from persistent failure

4. byted-volcengine-api-assistant

Responsibilities:

• Identify the product ServiceCode
• Search available OpenAPI
• Query resource details, configurations, and mapping relationships

Boundaries:

• It is a data supplementation capability, not a topology analysis engine
• It only works within the scope of OpenAPI capabilities

When it is suitable to call:

• The resource type cannot be directly determined from the alarm text
• The topology is missing listeners, backend groups, instances, or forwarding rule mappings
• Need to supplement product configurations, binding relationships, and health check details

When to trigger

When the following scenarios occur, this Skill should be actively triggered:

• The monitoring platform generates a single-point alarm
• An alarm storm occurs on the monitoring platform, and it is necessary to determine whether it belongs to the same chain issue
• On-duty personnel submit phenomena such as "a certain business is unavailable," "a certain domain name reports an error," or "a certain API times out" in the session
• Availability or performance anomalies occur shortly after a change
• The user wants to know which alarm in a group of alarms is closer to the root cause and which are only derived results

Even if the user does not explicitly say "topology" or "alarm merging," as long as the essence of the problem isconverging upstream and downstream anomalies around a fault phenomenon and locating the true root cause, it should be triggered.

Input information

Preferentially extract the following fields from alarm text, log excerpts, or free-form descriptions:

• Account
• Alarm policy
• Region Chinese name
• Region ID corresponding to the region
• Alarm level
• Project
• Alarm time
• Time window
• Cloud product
• Resource ID
• Instance ID
• Listener ID
• Domain name
• IP
• Current value
• Metric name
• Business phenomenon description
• Multiple related alarms appearing in the same session

If there are no explicit fields in the text, continue to identify:

• Product alias
• Resource hierarchy keywords
• Metric semantics such as health checks, connection counts, status codes, timeouts, and probe failures
• Context words such as entry layer, forwarding layer, compute layer, and network layer

If the resource ID is missing, it is also allowed to infer candidate resources from the domain name, IP, or phenomenon description, but the final answer must explicitly mark the uncertainty.

Standard execution process

Step 1: Structured input

First organize the raw input into a structured object, for example:

{
  "account": "xx",
  "policy_name": "xx",
  "region_name": "华北2（北京）",
  "region_id": "cn-beijing",
  "severity": "严重",
  "project": "mysite",
  "alarm_time": "2026-05-09 17:26:11",
  "time_window": "alarm_time +/- 15m",
  "product": "clb",
  "resource_id": "clb-xxx",
  "instance_id": "",
  "listener_id": "lsn-xxx",
  "domain": "",
  "ip": "",
  "metric_name": "监听器/后端服务器异常个数",
  "current_value": "AVG(xxx)[1m]:1Count",
  "symptom": "",
  "related_alerts": []
}

If some fields are missing, explicitly mark them as "missing information"; do not fill them in on your own.

Step 2: Identify the primary alarm node

First determine which layer the primary origin of this fault is closer to:

• Entry layer:domain、eip、clb、alb、listener
• Forwarding layer:rule、server_group
• Compute layer:ecs
• Network layer:security_group、subnet、vpc
• Other product layers: need to use API to further confirm

Judgment principles:

• If the alert contains both 资源 ID、实例 ID、监听器 ID, distinguish between the “alert object” and the “potential root cause object”
• Domain names and entry alerts are usually only entry layer signals and should not be directly equated with the final root cause
• If it is a load balancer type of alert, you must continue tracing the listener, backend group, and instances

Step 3: Obtain the fault topology context

Obtain the upstream and downstream topology around the primary alert node. The priority is as follows:

1. Prefer to use topology data that already exists in the workspace and can be reused
2. If the existing topology does not cover the target resource, call byted-volcengine-topology-builder
3. Use byted-volcengine-topology-analyzer to expand the upstream and downstream of the target node, the entry link, and the affected resources

The goal of this step is not “drawing a diagram for display”, but to establish an analyzable fault context for subsequent alert merging.

Step 4: Check related anomalies within the same topology

Within the already expanded topology scope, check whether other resources also have abnormal signals within the same time window.

Sources of anomalies may include:

• Multiple alerts provided in the current session
• Other monitoring metrics on the same resource link
• Via byted-volcengine-cloudmonitor Synchronous anomalies queried
• Abnormal status fields obtained through API supplementary data retrieval

If the alert platform cannot be queried directly at present, it should degrade to:

• Analyze the existing alert set in the session
• Analyze the real-time status of the current primary resource and its upstream and downstream
• Clearly state that “it is currently impossible to fully pull all alerts in the same time window”

Step 5: Merge alerts and classify them

After obtaining anomalies within the same topology, they must be divided into at least three categories:

• root_alert: alerts that are closer to the link convergence point and can explain the propagation of other anomalies
• derived_alert: derived alerts formed by propagation of the root problem to upstream or downstream
• noise_alert: alerts that are weakly related to the current primary link, have insufficient evidence, or cannot be proven to be homologous

Merging principles:

• Prioritize finding the node or resource layer where multiple abnormal paths ultimately converge
• Prioritize finding nodes that have both topological explanatory power and runtime anomaly evidence
• Do not forcibly group alerts that are merely close in time but unrelated in the link

Step 6: Supplement data and verify as needed

In the following situations, use supplementary capabilities as needed:

• There are nodes in the topology but the relationships are incomplete: call byted-volcengine-api-assistant
• Need to confirm whether the anomaly is still ongoing or whether it is simultaneously amplifying: call byted-volcengine-cloudmonitor
• Cannot directly map from domain name or IP to resource: prioritize using topology and API to infer

Usage principles:

• API and monitoring are for supplementing evidence and do not replace topology context
• When there is no evidence, do not pretend that “merging has been completed”
• When merging fails, it is better to be conservative than to misjudge noise alerts as having the same root cause

Step 7: Form a ranking of root cause candidates

After integrating the following evidence, then provide root cause candidates:

• Facts from the original alert text
• Location in the topology and upstream/downstream relationships
• Abnormal aggregation within the same topology
• Resource details and binding relationships obtained through API data completion
• Synchronous anomalies or persistent anomalies in monitoring time series

At minimum, the following must ultimately be answered:

• Which resource is the primary node for this fault analysis
• Which alerts are merged into the same issue
• Which node is most likely the root cause, and which is merely a propagation result
• What is the current scope of impact
• Which node should be handled first

Root cause determination principles

1. Do not directly equate the earliest occurring alert with the root cause

The earliest triggered alert is not necessarily the closest to the fault source; it may simply be a more sensitive entry signal.

2. Prioritize finding the link convergence point

If multiple abnormal paths ultimately converge on the same resource, the same layer of components, or the same resource group, it should be prioritized as a high-confidence candidate root cause.

3. Distinguish between "cause" and "propagation result"

For example:

• An upstream domain error may be only a result
• An entry load balancer health check anomaly may be only an intermediate signal
• An anomaly in a backend instance, server group, or network boundary is more likely to be close to the true root cause

4. Alert merging must be evidence-based

At least one of the following must be satisfied to merge with high confidence:

• There is a clear dependency relationship in the topology
• Synchronous anomalies appear within the same time window
• API or monitoring status can corroborate each other

5. Root cause candidates must carry confidence levels

It is recommended to express them at the following levels:

• confirmed: Supported by clear links and runtime evidence
• high: Topology and monitoring are highly consistent, but a small amount of detail is still missing
• medium: The link is reachable, but the evidence is not sufficiently complete
• low: Can only be localized to a certain layer, and the unique node cannot be confirmed

Response structure

When outputting, prioritize using the following structure:

结论：
- 当前最可能的根因候选是 ...
- 置信度是 ...
- 当前优先处理节点是 ...

告警归并结果：
- 主告警: ...
- 已归并的派生告警: ...
- 识别为噪声的告警: ...

拓扑与影响：
- 故障入口: ...
- 中间链路: ...
- 下游关键资源: ...
- 当前影响范围: ...

根因候选排序：
- 候选 1: ... | 证据: ...
- 候选 2: ... | 证据: ...
- 候选 3: ... | 证据: ...

事实 / 推断 / 待确认：
- 已确认事实: ...
- 高置信推断: ...
- 待确认项: ...

建议动作：
- 先处理 ...
- 再验证 ...
- 如需升级，交接摘要如下 ...

Expected output

When using this Skill, the final output should cover the following as much as possible:

• Ranking of root cause candidates and evidence chain
• Alert merging results and impacted topology view
• Current fault impact scope, priority handling nodes, and recommended remediation actions
• Structured diagnostic summary usable for escalation handover

Structured diagnostic summary requirements

If the user needs rapid escalation or handover, at least the following should be provided:

• Fault title
• Main affected business or entry point
• Current primary root cause candidate
• Number of merged alerts
• Current impact scope
• Priority handling node
• Verified evidence
• Items pending confirmation
• Recommended next actions

Troubleshooting recommendation generation rules

Troubleshooting recommendations should cover at least three categories:

1. Immediate verification items

• Whether the current primary root cause candidate is still continuously abnormal
• Whether there are still more resources with synchronous anomalies in the same topology
• Whether the priority handling node has already experienced health check failures, abnormal status, or capacity degradation

2. Link verification items

• Whether the upstream entry point is still reachable
• Whether the binding relationships of intermediate forwarding, listeners, backend groups, or instances are abnormal
• Whether there are link single points or redundancy failures

3. Mitigation and handover items

• Whether the current suspected faulty node can be bypassed
• Whether the abnormal node can be temporarily removed or traffic can be switched back to the standby entry point
• If escalation is needed, first hand over the primary root cause candidates, impact scope, and verified evidence

Failure fallback strategy

Case 1: Unable to identify the product or resource

Handling method:

• First infer candidate resources from the domain name, IP, resource ID, or instance ID
• Then call byted-volcengine-api-assistant to supplement product and API information
• If it still cannot be confirmed, clearly state "the resource type currently cannot be uniquely identified"

Case 2: Topology missing or match failure

Handling method:

• Call byted-volcengine-topology-builder to rebuild
• If there is still no match, it means the current topology does not cover the target resource
• Do not hard-code upstream and downstream relationships; directly state that the link evidence is insufficient

Case 3: Unable to fully retrieve alarms in the same time window

Handling method:

• Fall back to the alarms already available in the session and the current real-time evidence for partial correlation
• Clearly state that "all alarms in the same time window currently cannot be fully retrieved, and the correlation results may be incomplete"

Case 4: Monitoring or API data supplementation fails

Handling method:

• Fall back to the existing topology and original alarm text for conservative judgment
• Clearly state which type of dynamic evidence is missing
• Lower the confidence of the root cause candidates

Not applicable scenarios

Do not treat this Skill as a complete solution in the following scenarios:

• The user only wants to query a single monitoring metric
• The user only wants to query which OpenAPIs a certain product supports
• The user only wants to build a graph and does not care about alarm correlation and root cause localization
• Account permissions are missing, so topology, monitoring, and APIs cannot be queried
• The user explicitly requests execution of a playbook, but authorized mode has not been entered and there is no automatic execution capability

Example scenario

监控平台连续产生多条告警：
- 域名 5xx 升高
- CLB 监听器后端异常个数 > 0
- 单台 ECS 探活失败

值班人员在群里补充：
- mysite 首页访问报错
- 发生时间与告警时间一致

Recommended handling order:

1. First extract the domain name, entry resource, instance ID, listener ID, and time window
2. Expand the upstream and downstream topology around the primary resource
3. Check whether there are synchronous anomalies within the same topology
4. Distinguish root alarms, derived alarms, and noise alarms
5. Rank the most likely root cause candidates
6. Output the impact scope, priority handling nodes, and a structured diagnostic summary

Final answer constraints

When using this Skill, the final answer must comply with the following constraints:

• Give the conclusion first, then the evidence
• Clearly distinguish "fact / inference / to be confirmed"
• Do not fabricate business names, domain names, or service names
• Do not present static topology inferences as real-time facts
• Do not forcibly correlate alarms that are close in time but unrelated in the link
• If the correlation scope is incomplete, clearly acknowledge the boundary