Overview
This tool safely performs a DCS (Redis) master-slave failover drill on Huawei Cloud, verifying instance HA capability under AZ power-off (master node failure) scenarios. It runs a three-phase pipeline:
prepare (checks + risk assessment) -> execute (failover) -> report (generate report)
- prepare: query DCS instance, check IAM permissions, architecture hard-gate (ha + cross-AZ + RUNNING), generate experiment config directory, and run risk assessment
- execute: load config, safely execute master-slave failover (
ChangeMasterStandby), poll for completion, data consistency check - report: read config and execution result, generate comprehensive HTML report (instance info, safety checks, topology change, failover timeline, data consistency check)
Triggers / Trigger Conditions
Triggered when the user needs to perform a DCS master-slave failover, switchover, fault drill, AZ power-off drill, or failover experiment.
Triggers: DCS primary/standby switchover, primary/standby switchover drill, DCS failure drill, primary/standby switchover, switchover experiment, dcs failover, DCS failover, high availability drill, high availability verification, DCS HA switchover, AZ power outage drill, primary/standby swap, DCS switchover, Redis failure drill, DCS disaster recovery drill
Trigger conditions:
- User mentions DCS master-slave failover or failover operation
- User needs to perform master-slave switchover on a ha-type DCS instance
- User needs a DCS fault drill or HA verification (AZ power-off scenario)
- User needs to swap master-slave or switchover on a DCS instance
- User needs to verify DCS HA / disaster recovery capability
Prerequisites
| Condition | Description | Verification |
|---|---|---|
| hcloud CLI | Installed and authenticated (KooCLI >= 3.0.0) | hcloud version |
| AK/SK | Configured via hcloud profile. Script only checks if read-only calls succeed; actual auth is handled by hcloud profile | hcloud DCS ListInstances --cli-region=cn-south-1 |
| Python 3.8+ | To run scripts | python3 --version |
| Master-slave DCS instance | A ha-type instance exists in the target region | hcloud CLI |
| Instance status RUNNING | Instance must be running before failover | prepare phase auto-checks |
Configuration
Shared config file config/config.json:
{
"region": "cn-south-1",
"instance_id": "<your-dcs-instance-id>"
}
| Field | Required | Description |
|---|---|---|
region |
Yes | Huawei Cloud region ID (e.g. cn-south-1, cn-north-4) |
instance_id |
Yes | Target DCS instance ID |
⚠️ You must modify
config/config.jsonbefore running: replaceregionandinstance_idwith your own region and DCS instance ID. The default value is a placeholder; using it directly will fail.
ℹ️ No need to adjust hcloud env config: This skill does not require or recommend modifying hcloud profile (e.g. readTimeout). The script layer uses a sufficiently long subprocess timeout (120s) for CLI calls, and
run_hclouddetects CLI timeout signatures (调用API超时/context deadline exceeded/readTimeout), applying idempotent protection for synchronous mutation API timeouts (read-only recheck, no command replay). User hcloud env config can stay as-is.
Parameter Confirmation
Required Parameters
- region: Huawei Cloud region ID (e.g. cn-south-1, cn-north-4), configured in
config/config.json - instance_id: Target DCS master-slave instance ID, configured in
config/config.json
Optional Parameters
- --phase: Pipeline phase to run (prepare / execute / report / all), default
all - --force: Skip critical risk blocking and force failover execution (requires manual confirmation)
- --config-dir: Config directory path, default
experiments/
Conversational Usage (Recommended)
The user simply says "execute DCS primary/standby switchover" (or an equivalent trigger), and the agent guides through the drill automatically. The user does not need to manually edit any config file.
Agent Guidance Flow
- Probe instances: use
hcloud DCS ListInstances --cli-region=<region>to query instances in the target region; if the user hasn't specified a region, probe by priority (cn-south-1 / cn-north-4 / cn-north-1 / cn-east-3) to find a drillable master-slave instance (ha + cross-AZ + RUNNING) - Confirm target: show candidate instances to the user, confirm the target (if only one candidate, can confirm directly)
- Confirm execution: confirm user authorizes a real failover (failover is a mutation that swaps master-slave roles, must be confirmed). Options should only revolve around "execute/cancel", do not offer "prepare-only" options — the user's goal is to execute failover; prepare is just an internal pipeline phase
- Write config: write the confirmed region / instance_id into
config/config.json - Execute full pipeline: internally call
python3 scripts/run.pyto perform the real failover (if user confirms forced drill, internally append--force, no need to expose CLI args to user) - Present results: present CLI output as-is to the user, point out the report path
Questions for Missing Information
| Missing info | Question |
|---|---|
| Target instance | "Under the current account, <region> has instance X; confirm which one to use?" |
| Region | "Please provide the target region (e.g., cn-south-1 South China-Guangzhou)" |
| Execution confirmation | "Confirm executing primary/standby switchover on instance X? (This will switch the primary/standby roles, causing a brief connection interruption)" |
⚠️ Questions must only revolve around "execute/cancel". Never offer "prepare-only" or "risk-assessment-only" options — those are invalid questions that treat internal pipeline phases as standalone goals.
Key Interaction Rules
- Failover is a mutation: must confirm with user before executing, never default-execute
- Risk disclosure: when prepare detects risks (including critical), must fully show each risk's impact and suggestion to the user
- Critical risk secondary confirmation: when prepare detects critical risks, must show risks and confirm whether to proceed, never auto-proceed. Confirmation options are "Force drill / Cancel drill" (after user confirms forced drill, agent internally appends
--force, not exposed to user), must not offer "prepare-only" invalid options - DCS.4104 cooldown: if failover is rejected (instance in recovery), clearly tell the user "in failover cooldown, retry in a few minutes"
- Idempotent protection: if instance is already in target state (e.g. already failed over last round), inform user of current role status to avoid redundant failover
Quick Start
One-Click Full Pipeline
# Full pipeline (skill invocation performs real failover)
python3 scripts/run.py
Phase-by-Phase Execution
# 1. Prepare checks
python3 scripts/run.py --phase prepare
# 2. Execute failover
python3 scripts/run.py --phase execute
# If prepare detected critical risks and user confirmed to proceed, agent internally appends --force (not exposed to user):
# python3 scripts/execute.py --config-dir experiments --force
# 3. Generate report
python3 scripts/run.py --phase report
Three Phases
Phase 1: prepare (Preparation Checks)
Query DCS resources and user permissions, generate config directory, and run architecture hard-gate + risk assessment.
- Query target DCS instance (ListInstances + ShowInstance to fill in cache_mode, etc.)
- Query master-slave node info (ShowNodesInformation)
- Check IAM permissions (hcloud auth readiness, DCS read/write permissions, failover permission)
- Generate config files (experiment.json, iam_policy.json, monitoring.json)
- Architecture hard-gate (11 checks): ha type, RUNNING, master-slave nodes exist, cross-AZ, etc.; required items block on failure
- Risk assessment (three-step pipeline):
- Detect: read-only probe of instance status, master-slave node health, backup freshness, CES load metrics (CPU/memory/connections/latency)
- Assess: grade each item by threshold (info/unknown/warning/critical), aggregate overall risk level (low/medium/blocked) with action recommendation
- Block: results written to
risk_assessment.jsonand embedded in readiness report; execute phase reads this file, blocks failover by default if critical items exist (user must confirm to proceed)
Output: config files and readiness report in experiments/ directory (including risk_assessment.json)
Phase 2: execute (Failover Execution)
⚠️ Mutation — master-slave failover actually swaps master-slave roles.
- Load config, pre-check (instance exists, ha, RUNNING, IAM permissions)
- Read
risk_assessment.jsonand show risk assessment results; block failover if critical items exist and user hasn't confirmed - Validate risk assessment consistency: block by default if
risk_assessment.jsonregion/instance_id doesn't match config.json (prevents "swap instance, use stale assessment to bypass") - Real-time recheck before failover: immediately recheck instance status and node health before failover (prepare snapshot may be stale); block by default if critical anomaly found
- Call
hcloud DCS ChangeMasterStandbyto execute master-slave failover - Poll node status every 5 seconds, wait for master-slave role swap to complete (timeout 120 seconds)
- Data consistency check (master node moved to slave AZ, node_id change, key count)
- Save execution result to
execution_result.json
User interaction: failover is a real mutation; execute performs a real failover (no dry-run). Critical risks block by default; user must confirm to proceed. User-facing options are only "Force drill / Cancel drill" (internally via --force, not exposed to user).
Phase 3: report (Report Generation)
Read config and execution result, generate comprehensive HTML report.
Report includes: result banner, instance info, safety checks, master-slave topology change, failover timeline, data consistency check.
🔒 HTML encoding requirement (prevent stored XSS): all external data from cloud APIs (instance names, node info, check items, etc.) must be HTML-escaped via
html.escapebefore insertion. Bothscripts/report.pyandscripts/prepare.pyapplyesc()/_esc()to all external data.
Output: templates/failover_report_YYYYMMDD_HHMMSS.html (standalone HTML, open in browser)
Risk Assessment
The prepare phase runs risk assessment in addition to architecture hard-gate checks. Assessment has three steps: detect -> assess -> block.
Detection Items (Read-Only Real-Time Probe)
| Dimension | Data source (read-only API) | Metric |
|---|---|---|
| Instance health | DCS ShowInstance |
Instance status RUNNING |
| Master-slave nodes | DCS ShowNodesInformation |
Master/slave node existence and status |
| Backup safety net | DCS ListBackupRecords |
Latest successful backup time, failed tasks in last 7 days |
| Load | CES ShowMetricData |
CPU/memory/connections/latency average |
| Deployment | ShowInstance az_codes |
Master-slave cross-AZ |
Grading Rules (Threshold-Based)
Each detection item is mapped to one of four levels: info / unknown / warning / critical. Any critical item -> overall risk blocked; warning or unknown items -> medium; otherwise -> low. Threshold table and business semantics: see references/risk-model.md.
Blocking Mechanism
- Console: prepare / execute phases prominently output overall level and each warning's impact and suggestion
- Report: risk section embedded in
readiness_report.html/readiness_report.md - Execution block:
executephase readsrisk_assessment.json, blocks failover by default if critical items exist (unless user confirms to proceed) - Anti-mismatch: reject if risk assessment doesn't match the target instance; anti-stale-snapshot: real-time recheck of instance and node status before execution
⚠️ Missing data is never misjudged, but not auto-passed: when an API is unavailable (e.g. no permission, no CES data), the item is marked "unknown" with an explanation, never fabricating a critical risk; but unknown items raise overall risk to "medium", prompting manual confirmation before execution.
CES Monitoring Metrics
During prepare phase risk assessment, the last 1 hour of CES data is queried. The monitoring config defines the following 8 DCS metrics (namespace SYS.DCS, dimension dcs_instance_id):
| Logic name | metric_name | Description | Unit |
|---|---|---|---|
| dcs_cpu_util | cpu_usage | CPU usage | % |
| dcs_mem_util | memory_usage | Memory usage | % |
| dcs_conn_usage | connections_usage | Connection usage | % |
| dcs_ops | instantaneous_ops | Instantaneous ops per second | count |
| dcs_connected_clients | connected_clients | Connected clients | count |
| dcs_command_max_rt | command_max_rt | Max command latency | us |
| dcs_in_flow | instantaneous_input_kbps | Instantaneous input flow | KiB/s |
| dcs_out_flow | instantaneous_output_kbps | Instantaneous output flow | KiB/s |
Composite Scoring
| Dimension | Weight | Basis |
|---|---|---|
| Failover | 40% | Failover success / duration (>30s deducts 10 per 10s) |
| Data | 30% | Consistency check pass ratio |
| Business | 30% | Default 100 (reserved for CES data deduction) |
Weighted total grade: >=85 excellent / >=70 good / >=60 fair / <60 poor
📌 Scoring is computed by the execute phase and written to
execution_result.json(scoresfield); the HTML report does not currently render a scoring section.
IAM Permission Requirements
| Permission | Description | Required |
|---|---|---|
dcs:instance:list |
List DCS instances | ✅ |
dcs:instance:get |
Get DCS instance detail | ✅ |
dcs:instance:getNodes |
Get master-slave node info | ✅ |
dcs:instance:modify |
Execute DCS master-slave failover (ChangeMasterStandby) | ✅ |
dcs:backup:list |
List backup records (risk assessment) | ✅ |
ces:metricData:get |
Query CES metric data | optional |
📌 Full IAM policy template (with Resource URN scope) see references/iam-policies.md.
Directory Structure
huawei-cloud-dcs-failover/
├── SKILL.md # This file
├── config/
│ └── config.json # Shared config (region + instance_id)
├── scripts/
│ ├── config.py # Config loading + run_hcloud + shared utils
│ ├── prepare.py # Phase 1: prepare checks (architecture hard-gate + risk assessment)
│ ├── risk.py # Risk assessment core module
│ ├── execute.py # Phase 2: execute failover (risk blocking, user confirm to proceed)
│ ├── report.py # Phase 3: generate report
│ └── run.py # Unified entry point (one-click full pipeline)
├── references/
│ ├── cli-installation-guide.md # hcloud CLI installation and auth guide
│ ├── iam-policies.md # IAM policy description
│ ├── acceptance-criteria.md # Acceptance criteria
│ ├── verification-method.md # Verification method
│ └── risk-model.md # Risk threshold table and business semantics
├── experiments/ # Intermediate artifacts (auto-cleaned before/after run)
│ ├── experiment.json
│ ├── iam_policy.json
│ ├── monitoring.json
│ ├── execution_result.json
│ ├── risk_assessment.json
│ ├── readiness_report.html
│ └── readiness_report.md
└── templates/ # Final deliverable directory (HTML report)
└── failover_report_YYYYMMDD_HHMMSS.html
Reference Documents
| Document | Description |
|---|---|
| references/cli-installation-guide.md | hcloud CLI installation and auth config guide |
| references/iam-policies.md | IAM policy template and config description |
| references/acceptance-criteria.md | Functional acceptance criteria and checklist |
| references/verification-method.md | Environment and functional verification, end-to-end flow |
| references/risk-model.md | Risk assessment threshold table and business semantics (risk model) |