SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillshuawei-cloud-rds-failover

huawei-cloud-rds-failover

Frontend & websites

View repository

Updated 10-15 22:15

About this skill

Huawei Cloud RDS Primary/Standby Switchover Drill

Overview

This tool is used to safely perform RDS primary/standby switchover (Failover) drills on Huawei Cloud. It is completed through a three-stage pipeline:

prepare（准备检查） → execute（执行倒换） → report（生成报告）
  • prepare: Query RDS HA instances, check IAM permissions, generate the experiment configuration directory, and performrisk assessment(see the dedicated section below)
  • execute: Read the configuration, perform the primary/standby switchover, poll and wait for completion, and collect logs and monitoring data
  • report: Read the configuration and execution results, and generate a comprehensive HTML report

Supports all RDS engines: MySQL, PostgreSQL, SQL Server, MariaDB, TaurusDB (GaussDB for MySQL).

Trigger Words/Trigger Conditions

This skill is triggered when the user needs to perform an RDS primary/standby switchover, primary/standby switch, fault drill, or failover experiment.

Trigger words: RDS primary/standby switchover, primary/standby switchover drill, failover, RDS fault drill, primary/standby switch, switchover experiment, rds failover, RDS failover, high availability drill, high availability verification, RDS HA switchover, primary/standby swap, RDS switchover, database fault drill, RDS disaster recovery drill

Trigger conditions：

  • The user mentions an RDS primary/standby switchover or failover operation
  • The user needs to perform a primary/standby switch on an HA-type RDS instance
  • The user needs an RDS fault drill or high availability verification
  • The user needs to perform a primary/standby swap or switchover on an RDS instance
  • The user needs to verify the RDS high availability disaster recovery capability

Prerequisites

Condition Description Verification method
hcloud CLI Installed and authenticated (KooCLI ≥ 3.0.0) hcloud version
AK/SK Configured through hcloud profile (hcloud configure init). The script only checks whether the AK/SK environment variablesexist(it does not read the actual values); actual authentication is completed by the hcloud profile hcloud RDS ListInstances --cli-region=cn-north-1
Python 3.8+ Run the script python3 --version
HA-type RDS instance At least one primary/standby instance exists in the target region View with hcloud CLI
Instance status is ACTIVE The instance must be running before the switchover Automatically checked in the prepare stage

Configuration

Shared configuration file config/config.json：

{
  "region": "<华为云区域ID>",
  "instance_id": "<目标RDS实例ID>"
}
Field Required Description
region Yes Huawei Cloud region ID (such as cn-north-1, cn-north-4)
instance_id Yes Target RDS instance ID (HA type, 32-digit hexadecimal + in01 suffix)

⚠️ Must be modified before running config/config.json: Replace region and instance_id with your own Huawei Cloud region and RDS instance ID. The default placeholder values cannot be used to run, and directly using someone else's instance ID may cause a primary/standby switchover to be performed on an unintended instance.

Quick Start

One-click complete process

# 预演（不实际执行倒换）
python3 scripts/run.py

# 实际执行完整流水线
python3 scripts/run.py --yes

Staged execution

# 1. 准备检查
python3 scripts/run.py --phase prepare
# 或直接运行
python3 scripts/prepare.py

# 2. 执行倒换（需人工确认）
python3 scripts/run.py --phase execute --yes
# 或直接运行
python3 scripts/execute.py --config-dir experiments --yes
# 若 prepare 检测出严重风险且已人工评估接受，可加 --force 强制演练（不推荐默认使用）：
#   python3 scripts/execute.py --config-dir experiments --yes --force

# 3. 生成报告
python3 scripts/run.py --phase report
# 或直接运行
python3 scripts/report.py --config-dir experiments

Three-stage description

Stage One: prepare (readiness check)

Query RDS resources and user permissions, generate the complete configuration directory, and completerisk assessment。

  • Query all RDS instances in the region (marker pagination retrieval, limit 50 pages/5000 records), filter HA type
  • Check IAM permissions (hcloud authentication readiness, RDS read/write permissions)
  • Generate configuration files (experiment.json, iam_policy.json, monitoring.json)
  • Run 15 preparation checks, generate HTML + Markdown preparation report
  • Risk assessment(three-step pipeline):
    • Detect: read-only probe of primary/standby replication status, replication latency (CES), storage water level, backup freshness, error logs from the last 24h, CPU/memory/disk/connection load, replication mode/switchover policy/max_connections
    • Grade: grade each item by threshold (notice/unknown/warning/critical), summarize the overall risk level (low/medium/block) and provide operation recommendations
    • Intercept: write results to risk_assessment.json and embed them in the preparation report; the execute phase reads this file, and when critical items exist, blocks the switchover by default (requires --force manual confirmation)

Output：experiments/ configuration files and preparation report in the directory (including risk_assessment.json）

Phase 2: execute (perform switchover)

⚠️ Change operation — primary/standby switchover will actually switch the primary and standby roles.

  • Load configuration, pre-check (instance exists, HA type, ACTIVE status, IAM permissions)
  • Read risk_assessment.json to display the risk assessment results; when critical risk items exist and --force is not addedblock switchover
  • Verify the risk assessment matches the instance：risk_assessment.json when the region/instance_id does not match config.json, block by default (to prevent "switching instances to get an old assessment approved")
  • Real-time re-check before execution: immediately re-check primary/standby replication status and replication latency before switchover (the prepare snapshot may be stale); if critical anomalies are found, block by default
  • Call hcloud RDS StartFailover to perform the primary/standby switchover
  • Poll the instance status every 5 seconds, waiting for the primary/standby role exchange to complete (timeout 120 seconds)
  • Collect RDS error logs, slow SQL logs, and CES monitoring metrics during the switchover
  • Save execution results to execution_result.json

Parameter：--yes means actual execution (default is dry run, no changes made)

Phase 3: report (generate report)

Read configuration and execution results, generate a comprehensive HTML report.

The report includes: result banner, instance information, security checks, IAM permissions, primary/standby topology changes, switchover timeline, CES monitoring metrics, error logs, slow SQL logs, alert rule status.

🔒 HTML encoding requirement (prevent stored XSS): all external data from cloud APIs in the report (instance names, error logs, slow SQL, alert information, risk detection items, etc.) must undergo html.escape entity encoding before being inserted into HTML. If logs or instance names returned by the cloud API contain malicious HTML/JS, inserting them directly without encoding will execute in the browser, causing stored XSS.scripts/report.py and scripts/prepare.py(preparation report) have both passed esc() / _esc() function to perform HTML escaping on all external data.

Output：report/failover_report_YYYYMMDD_HHMMSS.html(standalone HTML, open directly in browser)

Risk assessment

During the prepare phase, in addition to the 15 preparation checks, a risk assessment is also performed to help reduce the risk of anomalies during primary/standby switchover drills. The assessment has three steps:Detection → Grading → Interception。

Detection items (read-only real-time probes)

Dimension Data source (read-only interface) Metric
Instance health RDS ListInstances Whether instance status is ACTIVE
Primary/standby replication RDS ShowReplicationStatus Replication status normal / abnormal
Replication delay CES ShowMetricData rds073 Average delay over the last 1 hour
Storage watermark RDS ShowStorageUsedSpace + ListInstances volume.size Usage %
Backup fallback RDS ListBackups Time of most recent successful backup, failed tasks in the last 7 days
Error signals RDS ListErrorLogs Number of ERROR-level entries in the last 24h
Load CES ShowMetricData rds001/002/039/006 Average CPU/memory/disk/connection count
Configuration ListInstances + RDS ShowInstanceConfiguration Replication mode, switchover policy, max_connections, persistence parameters

Grading rules (threshold grading)

Each detection item is mapped to one of four levels by threshold: 🟢 Info / ⚪ Unknown / 🟠 Warning / 🔴 Critical. Any critical item → overall risk Blocked; if there are warning or unknown items → Medium; otherwise → Low. For the threshold table and business meaning, see references/risk-model.md。

Interception mechanism

  1. Console: During the prepare / execute phases, prominently output the overall level and the impact and suggestions of each warning
  2. Report: The risk section is embedded in readiness_report.html / readiness_report.md
  3. Execution interception：execute The phase reads risk_assessment.json, and if there is a critical item (and --force）By default, switchover execution is refused, manual confirmation is required to force the drill
  4. Prevent mismatches: If the risk assessment and the instance to be switched over do not match, release is refused;Prevent stale snapshots: Before execution, re-check replication status and replication delay in real time

⚠️ Missing data is not misjudged, but does not automatically release: When an interface is unavailable (for example, the engine does not support it, there is no permission, or CES has no data), that item is marked as "Unknown" with an explanation, and no critical risk is fabricated out of thin air; however, an unknown item will raise the overall risk to "Medium", prompting manual confirmation before execution. ⚠️ Forced drills have a cost：--force It is only used when you have manually assessed and accepted the risk; critical items (such as replication anomalies, full disks, or no backup) usually do indeed cause problems during switchover.

CES monitoring metrics

During switchover, 8 CES metrics are automatically collected (namespace SYS.RDS):

Script logical name MySQL/MariaDB/TaurusDB PostgreSQL SQL Server Description Unit
rds_cpu_util rds001_cpu_util rds001_cpu_util rds001_cpu_util CPU usage %
rds_mem_util rds002_mem_util rds002_mem_util rds002_mem_util Memory usage %
rds_disk_util rds039_disk_util rds039_disk_util rds039_disk_util Disk usage %
rds_connections_count rds006_conn_count rds042_database_connections rds054_db_connections_in_use Current connections count
rds_in_flow rds004_bytes_in rds004_bytes_in rds004_bytes_in Network input traffic Bytes/s
rds_out_flow rds005_bytes_out rds005_bytes_out rds005_bytes_out Network output traffic Bytes/s
rds_iops rds003_iops rds003_iops rds003_iops IOPS count/s
rds_replication_delay rds073_replication_delay rds046_replication_lag rds077_replication_delay Primary/standby replication delay s

Metric names adapt to the engine (by ces_config.py ENGINE_CES_CONFIG driver), the table above is kept in sync with the code. MySQL/MariaDB/TaurusDB share the same set of metric names; PostgreSQL and SQL Server have different metric names for connection count and replication latency.

IAM Permission Requirements

The authorization items have been verified one by one against the Huawei Cloud Identity Policy Authorization Reference . The authorization items in the "Alias" column are alternative spellings of the same permission—writing either the primary name or the alias in the policy is sufficient, and there is no need to grant both.

Permission Description Resource Type Required
rds:instance:listAll Query RDS instance list + storage space (ListInstances / ShowStorageUsedSpace); alias rds:instance:list - ✅
rds:instance:getReplicaStatus Query primary/standby replication status (ShowReplicationStatus); alias rds:instance:list instance ✅
rds:instance:getParameter Query instance parameter configuration (ShowInstanceConfiguration); alias rds:param:list instance ✅
rds:backup:list Query RDS backup list (ListBackups, risk assessment · backup freshness check) - ✅
rds:instance:switchover Execute RDS primary/standby switchover (StartFailover) instance ✅
rds:log:getErrorLogs Query RDS error logs (ListErrorLogs); alias rds:log:list instance Optional
rds:log:getSlowLogs Query RDS slow SQL logs (ListSlowLogs); alias rds:log:list instance Optional
ces:alarms:list Query CES alarm rules - Optional
ces:metricData:get Query CES monitoring metric data - Optional

📌 For the complete IAM policy template (including Resource URN scope restrictions), see references/iam-policies.md. Risk assessment depends on rds:instance:getReplicaStatus、rds:instance:getParameter、rds:backup:list and other queries for replication status, storage water level, parameter configuration, and backup freshness; lacking the corresponding permissions will cause the check to be downgraded to "unknown".

Directory Structure

huawei-cloud-rds-failover/
├── SKILL.md                          # 本文件
├── config/
│   └── config.json                   # 共享配置（region + instance_id）
├── scripts/
│   ├── config.py                     # 配置加载模块
│   ├── prepare.py                    # 阶段一：准备检查（含风险评估接入）
│   ├── risk.py                       # 风险评估核心模块
│   ├── ces_config.py                 # CES 监控维度/指标按引擎适配（MySQL/PostgreSQL/SQLServer/MariaDB/TaurusDB）
│   ├── execute.py                    # 阶段二：执行倒换（含风险拦截 --force）
│   ├── report.py                     # 阶段三：生成报告
│   └── run.py                        # 统一入口（一键全流程）
├── references/
│   ├── cli-installation-guide.md     # hcloud CLI 安装与认证指南
│   ├── iam-policies.md               # IAM 权限策略说明
│   ├── acceptance-criteria.md        # 验收标准
│   ├── verification-method.md       # 验证方法
│   └── risk-model.md                 # 风险阈值表与业务含义
├── experiments/                      # 生成的配置目录
│   ├── experiment.json
│   ├── iam_policy.json
│   ├── monitoring.json
│   ├── execution_result.json
│   ├── risk_assessment.json
│   ├── readiness_report.html
│   └── readiness_report.md
└── report/                           # 生成的 HTML 报告
    └── failover_report_YYYYMMDD_HHMMSS.html

Reference Documents

Document Description
references/cli-installation-guide.md hcloud CLI installation and authentication configuration guide
references/iam-policies.md IAM permission policy template and configuration instructions
references/acceptance-criteria.md Functional acceptance criteria and checklist
references/verification-method.md Environment and functionality verification methods, end-to-end verification process
references/risk-model.md Risk assessment threshold table and business implications (risk model)
Huawei Cloud RDS Product Documentation RDS Relational Database Service product introduction
RDS API Reference — StartFailover Primary/standby switchover API interface description
CES Monitoring Metrics — RDS RDS CES monitoring metric list