SuperAPI AI Directory

huawei-cloud-dcs-failover

General & other

View repository

Updated 10-30 04:53

About this skill

Huawei Cloud DCS Master-Slave Failover (AZ Power-Off) Drill

Overview

This tool safely performs a DCS (Redis) master-slave failover drill on Huawei Cloud, verifying instance HA capability under AZ power-off (master node failure) scenarios. It runs a three-phase pipeline:

prepare (checks + risk assessment) -> execute (failover) -> report (generate report)
  • prepare: query DCS instance, check IAM permissions, architecture hard-gate (ha + cross-AZ + RUNNING), generate experiment config directory, and run risk assessment
  • execute: load config, safely execute master-slave failover (ChangeMasterStandby), poll for completion, data consistency check
  • report: read config and execution result, generate comprehensive HTML report (instance info, safety checks, topology change, failover timeline, data consistency check)

Triggers / Trigger Conditions

Triggered when the user needs to perform a DCS master-slave failover, switchover, fault drill, AZ power-off drill, or failover experiment.

Triggers: DCS primary/standby switchover, primary/standby switchover drill, DCS failure drill, primary/standby switchover, switchover experiment, dcs failover, DCS failover, high availability drill, high availability verification, DCS HA switchover, AZ power outage drill, primary/standby swap, DCS switchover, Redis failure drill, DCS disaster recovery drill

Trigger conditions:

  • User mentions DCS master-slave failover or failover operation
  • User needs to perform master-slave switchover on a ha-type DCS instance
  • User needs a DCS fault drill or HA verification (AZ power-off scenario)
  • User needs to swap master-slave or switchover on a DCS instance
  • User needs to verify DCS HA / disaster recovery capability

Prerequisites

Condition Description Verification
hcloud CLI Installed and authenticated (KooCLI >= 3.0.0) hcloud version
AK/SK Configured via hcloud profile. Script only checks if read-only calls succeed; actual auth is handled by hcloud profile hcloud DCS ListInstances --cli-region=cn-south-1
Python 3.8+ To run scripts python3 --version
Master-slave DCS instance A ha-type instance exists in the target region hcloud CLI
Instance status RUNNING Instance must be running before failover prepare phase auto-checks

Configuration

Shared config file config/config.json:

{
  "region": "cn-south-1",
  "instance_id": "<your-dcs-instance-id>"
}
Field Required Description
region Yes Huawei Cloud region ID (e.g. cn-south-1, cn-north-4)
instance_id Yes Target DCS instance ID

⚠️ You must modify config/config.json before running: replace region and instance_id with your own region and DCS instance ID. The default value is a placeholder; using it directly will fail.

ℹ️ No need to adjust hcloud env config: This skill does not require or recommend modifying hcloud profile (e.g. readTimeout). The script layer uses a sufficiently long subprocess timeout (120s) for CLI calls, and run_hcloud detects CLI timeout signatures (调用API超时 / context deadline exceeded / readTimeout), applying idempotent protection for synchronous mutation API timeouts (read-only recheck, no command replay). User hcloud env config can stay as-is.

Parameter Confirmation

Required Parameters

  • region: Huawei Cloud region ID (e.g. cn-south-1, cn-north-4), configured in config/config.json
  • instance_id: Target DCS master-slave instance ID, configured in config/config.json

Optional Parameters

  • --phase: Pipeline phase to run (prepare / execute / report / all), default all
  • --force: Skip critical risk blocking and force failover execution (requires manual confirmation)
  • --config-dir: Config directory path, default experiments/

Conversational Usage (Recommended)

The user simply says "execute DCS primary/standby switchover" (or an equivalent trigger), and the agent guides through the drill automatically. The user does not need to manually edit any config file.

Agent Guidance Flow

  1. Probe instances: use hcloud DCS ListInstances --cli-region=<region> to query instances in the target region; if the user hasn't specified a region, probe by priority (cn-south-1 / cn-north-4 / cn-north-1 / cn-east-3) to find a drillable master-slave instance (ha + cross-AZ + RUNNING)
  2. Confirm target: show candidate instances to the user, confirm the target (if only one candidate, can confirm directly)
  3. Confirm execution: confirm user authorizes a real failover (failover is a mutation that swaps master-slave roles, must be confirmed). Options should only revolve around "execute/cancel", do not offer "prepare-only" options — the user's goal is to execute failover; prepare is just an internal pipeline phase
  4. Write config: write the confirmed region / instance_id into config/config.json
  5. Execute full pipeline: internally call python3 scripts/run.py to perform the real failover (if user confirms forced drill, internally append --force, no need to expose CLI args to user)
  6. Present results: present CLI output as-is to the user, point out the report path

Questions for Missing Information

Missing info Question
Target instance "Under the current account, <region> has instance X; confirm which one to use?"
Region "Please provide the target region (e.g., cn-south-1 South China-Guangzhou)"
Execution confirmation "Confirm executing primary/standby switchover on instance X? (This will switch the primary/standby roles, causing a brief connection interruption)"

⚠️ Questions must only revolve around "execute/cancel". Never offer "prepare-only" or "risk-assessment-only" options — those are invalid questions that treat internal pipeline phases as standalone goals.

Key Interaction Rules

  • Failover is a mutation: must confirm with user before executing, never default-execute
  • Risk disclosure: when prepare detects risks (including critical), must fully show each risk's impact and suggestion to the user
  • Critical risk secondary confirmation: when prepare detects critical risks, must show risks and confirm whether to proceed, never auto-proceed. Confirmation options are "Force drill / Cancel drill" (after user confirms forced drill, agent internally appends --force, not exposed to user), must not offer "prepare-only" invalid options
  • DCS.4104 cooldown: if failover is rejected (instance in recovery), clearly tell the user "in failover cooldown, retry in a few minutes"
  • Idempotent protection: if instance is already in target state (e.g. already failed over last round), inform user of current role status to avoid redundant failover

Quick Start

One-Click Full Pipeline

# Full pipeline (skill invocation performs real failover)
python3 scripts/run.py

Phase-by-Phase Execution

# 1. Prepare checks
python3 scripts/run.py --phase prepare

# 2. Execute failover
python3 scripts/run.py --phase execute
# If prepare detected critical risks and user confirmed to proceed, agent internally appends --force (not exposed to user):
#   python3 scripts/execute.py --config-dir experiments --force

# 3. Generate report
python3 scripts/run.py --phase report

Three Phases

Phase 1: prepare (Preparation Checks)

Query DCS resources and user permissions, generate config directory, and run architecture hard-gate + risk assessment.

  • Query target DCS instance (ListInstances + ShowInstance to fill in cache_mode, etc.)
  • Query master-slave node info (ShowNodesInformation)
  • Check IAM permissions (hcloud auth readiness, DCS read/write permissions, failover permission)
  • Generate config files (experiment.json, iam_policy.json, monitoring.json)
  • Architecture hard-gate (11 checks): ha type, RUNNING, master-slave nodes exist, cross-AZ, etc.; required items block on failure
  • Risk assessment (three-step pipeline):
    • Detect: read-only probe of instance status, master-slave node health, backup freshness, CES load metrics (CPU/memory/connections/latency)
    • Assess: grade each item by threshold (info/unknown/warning/critical), aggregate overall risk level (low/medium/blocked) with action recommendation
    • Block: results written to risk_assessment.json and embedded in readiness report; execute phase reads this file, blocks failover by default if critical items exist (user must confirm to proceed)

Output: config files and readiness report in experiments/ directory (including risk_assessment.json)

Phase 2: execute (Failover Execution)

⚠️ Mutation — master-slave failover actually swaps master-slave roles.

  • Load config, pre-check (instance exists, ha, RUNNING, IAM permissions)
  • Read risk_assessment.json and show risk assessment results; block failover if critical items exist and user hasn't confirmed
  • Validate risk assessment consistency: block by default if risk_assessment.json region/instance_id doesn't match config.json (prevents "swap instance, use stale assessment to bypass")
  • Real-time recheck before failover: immediately recheck instance status and node health before failover (prepare snapshot may be stale); block by default if critical anomaly found
  • Call hcloud DCS ChangeMasterStandby to execute master-slave failover
  • Poll node status every 5 seconds, wait for master-slave role swap to complete (timeout 120 seconds)
  • Data consistency check (master node moved to slave AZ, node_id change, key count)
  • Save execution result to execution_result.json

User interaction: failover is a real mutation; execute performs a real failover (no dry-run). Critical risks block by default; user must confirm to proceed. User-facing options are only "Force drill / Cancel drill" (internally via --force, not exposed to user).

Phase 3: report (Report Generation)

Read config and execution result, generate comprehensive HTML report.

Report includes: result banner, instance info, safety checks, master-slave topology change, failover timeline, data consistency check.

🔒 HTML encoding requirement (prevent stored XSS): all external data from cloud APIs (instance names, node info, check items, etc.) must be HTML-escaped via html.escape before insertion. Both scripts/report.py and scripts/prepare.py apply esc() / _esc() to all external data.

Output: templates/failover_report_YYYYMMDD_HHMMSS.html (standalone HTML, open in browser)

Risk Assessment

The prepare phase runs risk assessment in addition to architecture hard-gate checks. Assessment has three steps: detect -> assess -> block.

Detection Items (Read-Only Real-Time Probe)

Dimension Data source (read-only API) Metric
Instance health DCS ShowInstance Instance status RUNNING
Master-slave nodes DCS ShowNodesInformation Master/slave node existence and status
Backup safety net DCS ListBackupRecords Latest successful backup time, failed tasks in last 7 days
Load CES ShowMetricData CPU/memory/connections/latency average
Deployment ShowInstance az_codes Master-slave cross-AZ

Grading Rules (Threshold-Based)

Each detection item is mapped to one of four levels: info / unknown / warning / critical. Any critical item -> overall risk blocked; warning or unknown items -> medium; otherwise -> low. Threshold table and business semantics: see references/risk-model.md.

Blocking Mechanism

  1. Console: prepare / execute phases prominently output overall level and each warning's impact and suggestion
  2. Report: risk section embedded in readiness_report.html / readiness_report.md
  3. Execution block: execute phase reads risk_assessment.json, blocks failover by default if critical items exist (unless user confirms to proceed)
  4. Anti-mismatch: reject if risk assessment doesn't match the target instance; anti-stale-snapshot: real-time recheck of instance and node status before execution

⚠️ Missing data is never misjudged, but not auto-passed: when an API is unavailable (e.g. no permission, no CES data), the item is marked "unknown" with an explanation, never fabricating a critical risk; but unknown items raise overall risk to "medium", prompting manual confirmation before execution.

CES Monitoring Metrics

During prepare phase risk assessment, the last 1 hour of CES data is queried. The monitoring config defines the following 8 DCS metrics (namespace SYS.DCS, dimension dcs_instance_id):

Logic name metric_name Description Unit
dcs_cpu_util cpu_usage CPU usage %
dcs_mem_util memory_usage Memory usage %
dcs_conn_usage connections_usage Connection usage %
dcs_ops instantaneous_ops Instantaneous ops per second count
dcs_connected_clients connected_clients Connected clients count
dcs_command_max_rt command_max_rt Max command latency us
dcs_in_flow instantaneous_input_kbps Instantaneous input flow KiB/s
dcs_out_flow instantaneous_output_kbps Instantaneous output flow KiB/s

Composite Scoring

Dimension Weight Basis
Failover 40% Failover success / duration (>30s deducts 10 per 10s)
Data 30% Consistency check pass ratio
Business 30% Default 100 (reserved for CES data deduction)

Weighted total grade: >=85 excellent / >=70 good / >=60 fair / <60 poor

📌 Scoring is computed by the execute phase and written to execution_result.json (scores field); the HTML report does not currently render a scoring section.

IAM Permission Requirements

Permission Description Required
dcs:instance:list List DCS instances ✅
dcs:instance:get Get DCS instance detail ✅
dcs:instance:getNodes Get master-slave node info ✅
dcs:instance:modify Execute DCS master-slave failover (ChangeMasterStandby) ✅
dcs:backup:list List backup records (risk assessment) ✅
ces:metricData:get Query CES metric data optional

📌 Full IAM policy template (with Resource URN scope) see references/iam-policies.md.

Directory Structure

huawei-cloud-dcs-failover/
├── SKILL.md                          # This file
├── config/
│   └── config.json                   # Shared config (region + instance_id)
├── scripts/
│   ├── config.py                     # Config loading + run_hcloud + shared utils
│   ├── prepare.py                    # Phase 1: prepare checks (architecture hard-gate + risk assessment)
│   ├── risk.py                       # Risk assessment core module
│   ├── execute.py                    # Phase 2: execute failover (risk blocking, user confirm to proceed)
│   ├── report.py                     # Phase 3: generate report
│   └── run.py                        # Unified entry point (one-click full pipeline)
├── references/
│   ├── cli-installation-guide.md     # hcloud CLI installation and auth guide
│   ├── iam-policies.md               # IAM policy description
│   ├── acceptance-criteria.md        # Acceptance criteria
│   ├── verification-method.md        # Verification method
│   └── risk-model.md                 # Risk threshold table and business semantics
├── experiments/                      # Intermediate artifacts (auto-cleaned before/after run)
│   ├── experiment.json
│   ├── iam_policy.json
│   ├── monitoring.json
│   ├── execution_result.json
│   ├── risk_assessment.json
│   ├── readiness_report.html
│   └── readiness_report.md
└── templates/                        # Final deliverable directory (HTML report)
    └── failover_report_YYYYMMDD_HHMMSS.html

Reference Documents

Document Description
references/cli-installation-guide.md hcloud CLI installation and auth config guide
references/iam-policies.md IAM policy template and config description
references/acceptance-criteria.md Functional acceptance criteria and checklist
references/verification-method.md Environment and functional verification, end-to-end flow
references/risk-model.md Risk assessment threshold table and business semantics (risk model)