huawei-cloud-mrs-host-fault-diagnose

Huawei Cloud MRS Host Fault Diagnosis Skill

Overview

This skill diagnoses Huawei Cloud MRS (MapReduce Service) cluster faults. Given a service name and/or node name, it progressively localizes the root cause: quick log scan first, host troubleshooting when host issues are found, detailed investigation when no conclusion is reached.

Architecture: Caller (Agent) -> check_api_mode.py (Python, scripts/) determines the API mode -> either lakewatch_api_client.py -> LakeWatch API -> MRS cluster (node resource data, logs, MRS Manager proxy) or manager_api_client.py -> MRS Manager REST API (28443). Per-layer knowledge base (fault_layer/ + scenarios/ + propagation.md in lakewatch mode; fault_layer_manager/ + scenarios_manager/ + propagation_manager.md in manager mode) drives the diagnosis flow; per-component config under components/ is shared by both modes; three fault layers (host -> instance -> service) with propagation chain tracing.

Note on language: This SKILL.md and the documents under references/ are written in English per the repository spec. The knowledge base documents under fault_layer/, fault_layer_manager/, scenarios/, scenarios_manager/, components/, propagation.md, and propagation_manager.md are also in English. Commands and code blocks are English throughout.

Applicable Scenarios:

A service is reported unhealthy and the root cause must be localized
An instance is reported faulty on a specific node
A host is reported unreachable or abnormal
Progressive fault triage from quick scan to deep investigation

Typical Use Cases:

"Something is wrong with KrbServer, please help diagnose it" (service fault, no node specified)
"KrbServer on 8-5-225-6 is down" (instance fault, service + node specified)
"Something is wrong with 8-5-225-6" (host fault, node only)
"MRS cluster KrbServer failed to start, cluster ID xxx"
"DBService stopped abnormally, node 8-5-225-6"

Critical Constraints

Important constraints:

Read-only: This skill only runs information-gathering commands (view logs, query status, collect resource data). It MUST NOT run any start/stop, modify, or delete operations.

User confirmation for repair: The skill only provides executable repair suggestions; it MUST NOT directly execute any repair operation. All repair actions require user confirmation.

Strict execution: Diagnose strictly according to the knowledge base content under this skill directory. Fabricating diagnostic commands outside the knowledge base is prohibited.

Prerequisites

1. Python Requirements

Python >= 3.7
Dependencies: pyyaml (YAML parsing), cryptography (Windows AES password encryption only)
Linux uses CryptoAPI for password encryption (no cryptography dependency)
Verify installation: python3 --version (Linux) / python --version (Windows)

This skill does NOT require KooCLI (hcloud). It calls the LakeWatch API through scripts/lakewatch_api_client.py (lakewatch mode) or the MRS Manager REST API through scripts/manager_api_client.py (manager mode). For the client setup, see CLI Installation Guide.

2. LakeWatch Credential Configuration

A valid LakeWatch service account (username + password)
The password MUST be encrypted with --encrypt-password and stored in scripts/lakewatch_api_config.yaml (auth.encrypted_password). Never store the plaintext password.
Security Rules:
- Never expose the LakeWatch password in conversation or command output
- Never ask the user to input the plaintext password in conversation; use the interactive --encrypt-password flow
- The token is cached locally with owner-only file permissions (Win: %TEMP%\lakewatch_token\, Linux: /tmp/lakewatch_token/)

3. MRS Manager Credential Configuration (Manager Mode)

Manager mode is enabled when scripts/manager_api_config.yaml exists and auth.encrypted_password is set.

A valid MRS Manager account (username + password)
Configure the Manager floating IP in server.host (port default 28443). To obtain it, run grep float_ip /opt/huawei/Bigdata/om-server/OMS/workspace/conf/oms.ini on the OMS node, or ask the cluster administrator.
The password MUST be encrypted with python3 scripts/manager_api_client.py --encrypt-password and stored in scripts/manager_api_config.yaml (auth.encrypted_password). Never store the plaintext password.
Security Rules:
- Never expose the Manager password in conversation or command output
- Never ask the user to input the plaintext password in conversation; use the interactive --encrypt-password flow
- Windows AES ciphertext requires the .aes_key file to be migrated together to decrypt on another machine; Linux SCC ciphertext is not portable across clusters
See MRS Manager API Client for the full client usage.

4. Access Permissions

Lakewatch mode: Reachability to the LakeWatch service endpoint (configured in scripts/lakewatch_api_config.yaml server.host/port); the LakeWatch account must have permission to call the MRS Manager proxy and collect node resource/log data on the target cluster
Manager mode: The script runtime environment must be able to reach the Manager port 28443; the Manager account needs read permissions on alarm, host, instance, and log APIs
See IAM Policies for the access model and required roles

5. Dependent Skill: huawei-cloud-mrs-host-alarm-diagnose

This skill references the per-alarm diagnosis knowledge base from the huawei-cloud-mrs-host-alarm-diagnose skill (sibling directory under skills/bigdata/mrs/). When the fault diagnosis flow encounters a known alarm (12006/12007/25000/25500/27001), it loads the corresponding document: ../huawei-cloud-mrs-host-alarm-diagnose/alarms/<alarm_id>.md in lakewatch mode, or ../huawei-cloud-mrs-host-alarm-diagnose/alarm_manager/<alarm_id>.md in manager mode.

If the alarm skill exists, load the referenced document and follow its diagnosis flow
If NOT exist, inform the user and proceed with the generic fault diagnosis flow
The dependency is document-level reference only (loading markdown by relative path), NOT a direct skill call. Both skills share the same LakeWatch/Manager API clients and config format.

Command Format Standard

This skill uses the LakeWatch API client instead of KooCLI. The unified command format is:

# Linux
python3 <skill_dir>/scripts/lakewatch_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2'

# Windows
python <skill_dir>/scripts/lakewatch_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2'

Element	Rule	Example
`python3` / `python`	Linux uses `python3`, Windows uses `python`	`python3 lakewatch_api_client.py`
`-a, --api`	API name to call (defined in `lakewatch_api_config.yaml`)	`-a collect_alarm_node_res_data`
`-p, --param`	API parameter in `key=value` form, repeatable	`-p 'cluster_id=xxx'`
Quoting	Every `-p` value MUST be wrapped in single quotes to prevent shell parsing of `[] {} \| ()`	`-p 'keywords=["ERROR"]'`

Windows (PowerShell) quote rule: every " inside a value must be replaced with """ (including " inside [] and {}), otherwise the server returns {"message":"Unknown exception","success":false,"code":"500"}:

# Correct on Windows
-p 'keywords=["""ERROR"""]'
-p 'env={"""PID""":"""123"""}'

# Wrong on Windows (will fail)
-p 'keywords=["ERROR"]'

Linux (bash) quote rule: keep " as-is inside the value, wrap the whole value in single quotes:

# Correct on Linux
-p 'keywords=["ERROR","Exception"]'
-p 'env={"PID":"123"}'

For the full API catalog, parameters, and the token/encryption mechanism, see LakeWatch API Client.

MRS Manager API Client (Manager Mode)

When check_api_mode.py reports manager, use manager_api_client.py instead of the LakeWatch client. The unified command format is:

# Linux
python3 <skill_dir>/scripts/manager_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2' --json

# Windows
python <skill_dir>/scripts/manager_api_client.py -a <api_name> -p 'key1=value1' -p 'key2=value2' --json

Element	Rule	Example
`python3` / `python`	Linux uses `python3`, Windows uses `python`	`python3 manager_api_client.py`
`-a, --api`	API name to call (defined in `manager_api_apis/`)	`-a get_instances`
`-p, --param`	API parameter in `key=value` form, repeatable	`-p 'service_name=KrbServer'`
`--json`	JSON formatted output	`--json`
`--auth`	Auth mode: `basic` (default) or `cookie`	`--auth cookie`
Quoting	Same quote rules as the LakeWatch client (`'` wrapping; Windows `"` -> `"""`)	`-p 'keywords=["ERROR"]'`

For the full API catalog, authentication modes, metric names, and password encryption mechanism, see MRS Manager API Client.

Workflow

Step 0: Determine the API Mode

Run the mode check script to determine whether diagnosis is based on MRS Manager or LakeWatch:

Windows: python scripts/check_api_mode.py
Linux: python3 scripts/check_api_mode.py

The script checks whether scripts/manager_api_config.yaml exists and whether encrypted_password is filled in, and returns a JSON result:

{"mode": "manager", "reason": "..."}     // manager-based
{"mode": "lakewatch", "reason": "..."}   // lakewatch-based

Rules:

manager_api_config.yaml does not exist -> default lakewatch
File exists but encrypted_password is empty -> default lakewatch
File exists and encrypted_password is not empty -> manager

The mode determines which knowledge base directories to load throughout the workflow:

Mode	Command script	Fault layer	Scenarios	Propagation	Alarm docs (sibling skill)
lakewatch	`lakewatch_api_client.py`	`fault_layer/`	`scenarios/`	`propagation.md`	`../huawei-cloud-mrs-host-alarm-diagnose/alarms/`
manager	`manager_api_client.py`	`fault_layer_manager/`	`scenarios_manager/`	`propagation_manager.md`	`../huawei-cloud-mrs-host-alarm-diagnose/alarm_manager/`

In the rest of this SKILL.md, <FAULT_LAYER> denotes fault_layer (lakewatch) or fault_layer_manager (manager), <SCENARIOS> denotes scenarios (lakewatch) or scenarios_manager (manager), and <PROPAGATION> denotes propagation.md (lakewatch) or propagation_manager.md (manager).

Step 1: Determine Fault Entry

Extract fault information from the user input and determine the diagnosis entry:

User Description	Entry	Step 1 Action
Has `service_name`, no `node_name` (e.g. "Something is wrong with KrbServer")	Service fault	Check all instance statuses, find faulty instances
Has `service_name` + `node_name` (e.g. "KrbServer on 8-5-225-6 is down")	Instance fault	Directly check that instance
Has `node_name`, no `service_name` (e.g. "Something is wrong with 8-5-225-6")	Host fault	Check host status, then check instances on the host

Step 2: Locate the Fault Object

Mode note: The command blocks below show lakewatch-mode commands. In manager mode, use the corresponding manager_api_client.py commands — see Core Commands -> Manager Mode Commands and <SCENARIOS>/data_collection.md for the per-mode equivalents.

Entry A: Service Fault (has service_name, no node_name)

Load components/<service_name>.md for component config. Query OMS primary/standby nodes, check process on each node:

# lakewatch mode
python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

Manager mode equivalents: get_oms_info (OMS primary/standby nodes); get_host_process (process status).

Decision:

Result	Next Step
All node processes normal	Step 4 detailed investigation
Some node processes missing	Step 3 quick log scan (for faulty nodes)
API call failed (node unreachable)	Step 4 host troubleshooting

Entry B: Instance Fault (has service_name + node_name)

Load components/<service_name>.md. Directly check process on that node:

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

Manager mode equivalent: get_host_process (process status).

Decision:

Result	Next Step
Process normal	Step 4 detailed investigation
Process missing	Step 3 quick log scan
API call failed (node unreachable)	Step 4 host troubleshooting

Entry C: Host Fault (has node_name, no service_name)

Query OMS primary/standby nodes, query node IP, ping the faulty node from OMS active node:

# lakewatch mode
python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

# lakewatch mode
python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=<cluster_id>' \
  -p 'node_name=<node_name>'

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=ping-check' \
  -p 'env={"TARGET_IP":"<target_ip>"}' \
  -p 'node_name=<oms_active_node>'

Manager mode equivalents: get_oms_info (OMS primary/standby); get_hosts -p 'hostname=<node_name>' (node IP); check_remote (remote connectivity — no dedicated ping-check API).

Decision:

Result	Next Step
Ping failed	Step 4 host troubleshooting (network/hardware)
Ping succeeded	Check all component processes on the host, find faulty instances -> Step 3 quick log scan

Step 3: Quick Log Scan

For the faulty node, quickly scan three layers of logs (Controller -> NodeAgent -> component), looking for clear ERROR:

# lakewatch mode - Controller log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=/var/log/Bigdata/controller' \
  -p 'log_file_name=exe.log*' \
  -p 'keywords=["<service_name>","ERROR","fail","timeout","Exception"]' \
  -p 'log_type=local' \
  -p 'node_name=<oms_active_node>'

# lakewatch mode - NodeAgent script log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=/var/log/Bigdata/nodeagent/scriptlog' \
  -p 'log_file_name=*.log*' \
  -p 'keywords=["<service_name>","ERROR","fail","exit"]' \
  -p 'log_type=local' \
  -p 'node_name=<node_name>'

Manager mode equivalents (see <SCENARIOS>/data_collection.md): browse_log (Controller exe.log, NodeAgent script.log); start_log_search + get_log_search_progress (keyword search).

If service_name is known, also check the component's own log (path from components/<service_name>.md):

# lakewatch mode
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=<log_directory>' \
  -p 'log_file_name=<log_file_name>' \
  -p 'keywords=["ERROR","Exception","FATAL","fail","OOM"]' \
  -p 'log_type=local' \
  -p 'node_name=<node_name>'

Decision:

Log Result	Next Step
Clear ERROR (e.g. OOM/permission/port conflict/config missing)	Output root cause
Log shows node unreachable / Agent timeout	Step 4 host troubleshooting
Multiple faulty nodes on same host	Step 4 host troubleshooting
No clear conclusion	Step 4 detailed investigation

Step 4: Detailed Investigation

When the quick log scan yields no conclusion, collect complete data:

Load [Data Collection](/data_collection.md) to collect process/port/HA/resource/alarm/framework logs
Load [Instance Fault Diagnosis](<FAULT_LAYER>/instance_fault.md) for instance-level diagnosis (includes scenario identification)
If needed, load [Service Fault Diagnosis](<FAULT_LAYER>/service_fault.md) for service-level diagnosis
If host issue is found, load [Host Fault Diagnosis](<FAULT_LAYER>/host_fault.md) for host-level diagnosis

Step 5: Propagation Chain Tracing

Load Propagation Chain to trace the root cause propagation path and impact scope.

Step 6: Output Diagnosis Conclusion

## Diagnosis Result

| Item | Content |
|------|---------|
| Diagnosis time | [time] |
| Cluster ID | [cluster_id] |
| Faulty component | [service_name] |
| Faulty node | [node_name] |

### Diagnosis Process

| Step | Result |
|------|--------|
| Instance status | [which nodes normal/abnormal] |
| Quick log scan | [found/not found clear ERROR] |
| Host troubleshooting | [normal/abnormal: ...] |
| Detailed investigation | [process/port/HA/resource results] |

### Propagation Path

[root cause] -> [propagation] -> [symptom] (single-layer root cause if no propagation)

### Root Cause Analysis

**Root cause layer**: [host/instance/service]
**Root cause type**: [specific reason]

### Repair Suggestion

| Priority | Operation | Description | Needs user confirmation |
|----------|-----------|-------------|-------------------------|
| 1 | [operation] | [description] | Yes |

Core Commands

Query OMS Primary/Standby Nodes

python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id=<cluster_id>'

Query Node IP

python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=<cluster_id>' \
  -p 'node_name=<node_name>'

Collect Node Resource Data

# Process basic info
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":"<process_name>"}' \
  -p 'node_name=<node_name>'

# Port check
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=port-check' \
  -p 'env={"PORT":"<port>"}' \
  -p 'node_name=<node_name>'

# HA resource status
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=ha-resource-status' \
  -p 'node_name=<node_name>'

# Disk space / Memory / CPU load
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'strategy_name=disk-space' \
  -p 'node_name=<node_name>'

Supported strategy_name values include: system-load, memory-usage, disk-space, disk-io, network-io, file-handle, port-check, high-cpu-processes, high-memory-process, zombie-process, dns-check, network-connectivity-test, process-basic-info, process-file-descriptor, jstack-thread-dump, disk-health-check, disk-smart-info, ha-resource-status, omm-process-tree, and more. See LakeWatch API Client for the full list.

Collect Alarm Log Data

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=<alarm_time>' \
  -p 'log_directory=<log_directory>' \
  -p 'log_file_name=<log_file_name>' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local'

When the log time format is non-standard ISO (e.g. [2026-07-07 20:54:25,171]), pass time_pattern:

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=<cluster_id>' \
  -p 'alarm_time=2026/07/07 20:54:00 GMT+08:00' \
  -p 'log_directory=/var/log/Bigdata/omm/oms/pms' \
  -p 'log_file_name=pms*.log' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local' \
  -p 'time_pattern=^\[([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})||ymdHMS'

Proxy MRS Manager GET API

# Query cluster services
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/services'

# Query host processes
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/hosts/<node_name>/processes'

# Query active alarms
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=<cluster_id>' \
  -p 'target_url=api/v2/clusters/<cluster_id>/alarms'

target_url MUST NOT start with /. The proxy requires Agent >= 1.0.5 and reported OMS node info. Only GET is supported currently.

Manager Mode Commands (Manager Mode)

When check_api_mode.py reports manager, use manager_api_client.py for the equivalent queries:

# Query OMS primary/standby nodes
python3 manager_api_client.py -a get_oms_info --json

# Query cluster services
python3 manager_api_client.py -a get_cluster_services \
  -p 'cluster_id=<cluster_id>' --json

# Query host detail (disk/memory/CPU usage)
python3 manager_api_client.py -a get_host_detail \
  -p 'hostname=<node_name>' --json

# Query host process status
python3 manager_api_client.py -a get_host_process \
  -p 'hostname=<node_name>' --json

# Query service instances (HA status)
python3 manager_api_client.py -a get_instances \
  -p 'cluster_id=<cluster_id>' \
  -p 'service_name=<service_name>' \
  -p 'hostname=<node_name>' --json

# Query host monitor metrics (dev_ prefix)
python3 manager_api_client.py -a get_host_metrics \
  -p 'hostname=<node_name>' \
  -p 'metric_names=dev_cpu_surp_avg,dev_load_one_min' --json

# Check remote node connectivity (replaces ping-check/network-connectivity-test)
python3 manager_api_client.py -a check_remote \
  -p 'remote_ip=<target_ip>' \
  -p 'remote_port=22' \
  -p 'remote_user_name=omm' \
  -p 'remote_client_path=/opt/huawei/Bigdata/nodeagent' --json

# Browse a log file (file_name must be a full path)
python3 manager_api_client.py -a browse_log \
  -p 'hostname=<node_name>' \
  -p 'file_name=/var/log/Bigdata/controller/exe.log' \
  -p 'start_line=1' \
  -p 'end_line=500' \
  -p 'search=<service_name>' --json

# Search logs by keyword (returns task_id, then poll progress)
python3 manager_api_client.py -a start_log_search \
  -p 'cluster_id=<cluster_id>' \
  -p 'key_word=ERROR' \
  -p 'start_time=<alarm_time>' \
  -p 'end_time=<current_time>' \
  -p 'services=<component>:<service_name>:<role_name>' \
  -p 'min_log_level=WARN' --json

python3 manager_api_client.py -a get_log_search_progress \
  -p 'search_id=<task_id>' --json

start_log_search services format: component:service:role (e.g. HDFS:HDFS:NameNode); start_time/end_time format: yyyy-MM-ddTHH:mm:ss. See MRS Manager API Client for metric names and full parameter rules.

Parameter Confirmation

Parameter	Required/Optional	Description	Default
`cluster_id`	Required	MRS cluster ID	N/A
`service_name`	Conditionally required	Faulty component (required for service/instance fault entry)	N/A
`node_name`	Conditionally required	Faulty node (required for instance/host fault entry)	N/A
`alarm_time`	Optional	Fault occurrence time, format `yyyy/MM/dd HH:mm:ss GMT+X:XX`	Current time
`strategy_name`	Required by `collect_alarm_node_res_data`	Resource collection strategy (lakewatch mode)	N/A
`log_directory`	Required by `collect_alarm_log_data`	Log directory, must be under `/var/log/` (lakewatch mode)	N/A
`log_file_name`	Required by `collect_alarm_log_data`	Log file name, no path separators (lakewatch mode)	N/A
`keywords`	Required by `collect_alarm_log_data`	Log keyword filter, JSON array (lakewatch mode)	N/A
`log_type`	Required by `collect_alarm_log_data`	`local` or `hdfs` (lakewatch mode)	N/A
`time_pattern`	Optional	Non-standard log time regex, format `regex\|\|format` (lakewatch mode)	N/A
`target_url`	Required by `access_manager_get`	MRS Manager API path, must NOT start with `/` (lakewatch mode)	N/A
`metric_names`	Required by `get_host_metrics`	Comma-separated monitor metric names with `dev_` prefix (manager mode)	N/A
`key_word`	Required by `start_log_search`	Log keyword to search (manager mode)	N/A
`current_time`	Required by `start_log_search`	Current time, format `yyyy-MM-ddTHH:mm:ss` (manager mode)	N/A
`file_name`	Required by `browse_log`	Full log file path (manager mode)	N/A

Output Format

The diagnosis report is output in Markdown, containing:

Diagnosis result table: diagnosis time, cluster ID, faulty component, faulty node
Diagnosis process: step-by-step results (instance status, quick log scan, host troubleshooting, detailed investigation)
Propagation path: root cause -> propagation -> symptom (single-layer if no propagation)
Root cause analysis: root cause layer (host/instance/service) + root cause type
Repair suggestion table: priority, operation, description, needs-user-confirmation (all repair actions require user confirmation)

See the template in the Workflow -> Step 6 section.

Verification Method

See Verification Method for the installation, configuration, and function verification steps.

Best Practices

Determine entry first: Based on user-provided information (service_name, node_name), determine whether the entry is service fault, instance fault, or host fault before starting diagnosis.
Progressive investigation: Always start with quick log scan (Step 3); only escalate to detailed investigation (Step 4) when no clear conclusion is reached.
Substitute placeholders: Replace <cluster_id>, <alarm_time>, <node_name>, <target_ip>, <process_name>, etc. with actual user-provided values; never hardcode them.
Quote parameters: Always wrap -p values in single quotes; on Windows PowerShell, escape " as """ to avoid code:500 errors.
Time format: alarm_time must follow yyyy/MM/dd HH:mm:ss GMT+X:XX; for non-standard log time formats, pass time_pattern.
Summarize results: Use a summarization tool to condense command output before analysis; large raw outputs should not be analyzed directly.
Reflect after diagnosis: After completing the checks, reflect on whether the root cause is confirmed; if not, re-check for missed steps.
Read-only: All commands are read-only; repair steps are suggestions only and require user confirmation before execution.
Command failure handling: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.

References

Document	Description
CLI Installation Guide	Python dependencies and LakeWatch/Manager client setup
IAM Policies	LakeWatch/MRS Manager access model and required roles
Verification Method	Installation, configuration, and function verification
Acceptance Criteria	Pass/fail criteria for skill testing
Fault Diagnosis Workflow	Progressive fault diagnosis workflow design
LakeWatch API Client	Full LakeWatch API catalog, parameters, token and encryption mechanism
MRS Manager API Client	Full MRS Manager API catalog, authentication modes, metric names and encryption mechanism
Related Commands	Common LakeWatch/Manager API commands quick reference
huawei-cloud-mrs-host-alarm-diagnose (sibling skill)	Dependency: per-alarm diagnosis knowledge base (`../huawei-cloud-mrs-host-alarm-diagnose/alarms/<alarm_id>.md` in lakewatch mode, `alarm_manager/<alarm_id>.md` in manager mode). See Prerequisites section 5 for details.
Data Collection	Complete data collection flow, lakewatch mode (Step 4)
Data Collection (Manager)	Complete data collection flow, manager mode (Step 4)
Host Fault Diagnosis	Host layer diagnosis, lakewatch mode
Instance Fault Diagnosis	Instance layer diagnosis (includes scenario identification), lakewatch mode
Service Fault Diagnosis	Service layer diagnosis, lakewatch mode
Host Fault Diagnosis (Manager)	Host layer diagnosis, manager mode
Instance Fault Diagnosis (Manager)	Instance layer diagnosis (includes scenario identification), manager mode
Service Fault Diagnosis (Manager)	Service layer diagnosis, manager mode
Propagation Chain	Root cause propagation path tracing, lakewatch mode
Propagation Chain (Manager)	Root cause propagation path tracing, manager mode
Common Scenario	6-phase common diagnosis framework, lakewatch mode
Common Scenario (Manager)	6-phase common diagnosis framework, manager mode
`scenarios/<scenario>.md`	Scenario-specific checks, lakewatch mode (install/start/stop/uninstall/reinstall/reinstall_host/scale_out/scale_in)
`scenarios_manager/<scenario>.md`	Scenario-specific checks, manager mode (install/start/stop/uninstall/reinstall/reinstall_host/scale_out/scale_in)
`components/<service_name>.md`	Per-component configuration (process, port, log path, etc.) — shared by both modes
`components/_template.md`	Template for new component configuration

Notes

Security: This skill is read-only. It never exposes the LakeWatch or MRS Manager password; passwords are encrypted via --encrypt-password and stored in the corresponding config YAML. Repair steps are suggestions only.
No KooCLI: This skill does not use hcloud; it calls the LakeWatch API through lakewatch_api_client.py or the MRS Manager REST API through manager_api_client.py. Do not mix in hcloud commands.
Mode switching: Run check_api_mode.py (Step 0) to determine the mode. In manager mode use manager_api_client.py and the fault_layer_manager/ + scenarios_manager/ + propagation_manager.md knowledge base; in lakewatch mode use lakewatch_api_client.py and fault_layer/ + scenarios/ + propagation.md. Do not mix clients across modes.
Command failure: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.
Known limitations: The access_manager_get proxy only supports GET requests (PUT is not yet available on the Agent side); collect_alarm_log_data requires log_directory to be under /var/log/; some strategy_name values require extra env parameters; in manager mode browse_log requires the full log file path, start_log_search services must follow component:service:role, and get_alarms may return 500 on some Manager versions (fall back to Controller exe.log browsing).
Cross-skill dependency: This skill references alarm diagnosis documents from the huawei-cloud-mrs-host-alarm-diagnose skill (../huawei-cloud-mrs-host-alarm-diagnose/alarms/<id>.md in lakewatch mode, alarm_manager/<id>.md in manager mode). See Prerequisites section 5 for the dependency declaration and handling rules. If the alarm skill is not installed, inform the user and proceed with the generic fault diagnosis flow.

About this skill

Overview

Critical Constraints

Prerequisites

1. Python Requirements

2. LakeWatch Credential Configuration

3. MRS Manager Credential Configuration (Manager Mode)

4. Access Permissions

5. Dependent Skill: huawei-cloud-mrs-host-alarm-diagnose

Command Format Standard

MRS Manager API Client (Manager Mode)

Workflow

Step 0: Determine the API Mode

Step 1: Determine Fault Entry

Step 2: Locate the Fault Object

Entry A: Service Fault (has service_name, no node_name)

Entry B: Instance Fault (has service_name + node_name)

Entry C: Host Fault (has node_name, no service_name)

Step 3: Quick Log Scan

Step 4: Detailed Investigation

Step 5: Propagation Chain Tracing

Step 6: Output Diagnosis Conclusion

Core Commands

Query OMS Primary/Standby Nodes

Query Node IP

Collect Node Resource Data

Collect Alarm Log Data

Proxy MRS Manager GET API

Manager Mode Commands (Manager Mode)

Parameter Confirmation

Output Format

Verification Method

Best Practices

References

Notes