SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillshuawei-cloud-ecs-shutdown-experiment

huawei-cloud-ecs-shutdown-experiment

Deployment & operations

View repository

Updated 10-15 22:14

About this skill

Huawei Cloud ECS Shutdown Chaos Experiment

⚠️ This skill performs real ECS shutdown operations in Phase 2. Ensure target instances can tolerate downtime. Emergency rollback is always available.

Overview

Full lifecycle ECS shutdown fault injection experiment for Huawei Cloud chaos engineering, covering three phases:

Phase Name Action Key Output
1 Prepare Discover, validate, generate config experiment.json + rollback_experiment.sh
2 Execute Shutdown → hold → rollback → verify execution-log.json + execution-report.md
3 Analyze Collect CES metrics + LTS logs, analyze patterns log-analysis-result.json + log-analysis-report.md

Scope: standalone ECS instances in any availability zone (non-CCE).

Prerequisites

  • hcloud CLI (KooCLI) — installed and configured with AK/SK. See references/cli-installation-guide.md.
  • Python 3 — for discovery, validation, execution, and analysis scripts.
  • Environment variables: HW_ACCESS_KEY, HW_SECRET_KEY, HW_REGION_NAME (optional, default cn-north-4).
  • IAM permissions: varies by phase — see references/iam-policies.md for the full matrix.
  • Phase 3 only: LTS log group ID (find via hcloud LTS ListLogGroups or LTS console).

Environment check (run before any phase):

bash scripts/check_env.sh --cli-region <region>

Phase 1: Prepare

Does NOT start the experiment. Only discovers targets, validates compatibility, and generates configuration files.

Step 1.1: Discover Target ECS Instances

python3 scripts/discover_ecs.py --cli-region <region>
python3 scripts/discover_ecs.py --cli-region <region> --az cn-north-4a --status ACTIVE
python3 scripts/discover_ecs.py --cli-region <region> --name-pattern "web-"

Output: JSON with instance ID, name, status, AZ, flavor, charging mode, and tags.

Step 1.2: Validate Compatibility [CRITICAL GATE]

python3 scripts/validate_targets.py --instances-file instances.json --cli-region <region>

Validation rules (see references/ecs-validation-rules.md for details):

  1. Status must be ACTIVE — can only shut down running instances
  2. Spot/bidding instance warning — may be released instead of stopped
  3. AS group association — shutdown may trigger auto-scaling activity
  4. Single-AZ risk — no cross-AZ redundancy if all targets in one AZ

Decision: errors → stop and fix; warnings only → review with user before proceeding.

Step 1.3: Generate Experiment Configuration

python3 scripts/generate_experiment.py \
    --targets "i-xxx,i-yyy" --cli-region <region> --duration 300 --output-dir ./experiments

Output directory: {timestamp}-ecs-shutdown-{slug}/ containing experiment.json + README.md. See references/experiment-template-guide.md for schema.

Step 1.4: Deploy Experiment

bash scripts/deploy_experiment.sh --experiment-dir ./experiments/{dir}/ --cli-region <region>

Generates rollback_experiment.sh (emergency rollback script) in the experiment directory.

Step 1.5: Review

Experiment is prepared but NOT started. Review experiment.json and README.md, then proceed to Phase 2 when ready.

Phase 2: Execute

Real shutdown operation. Target ECS instances are stopped. Ensure experiment.json from Phase 1 is available.

Step 2.1: Execute the Experiment

# Real execution (--yes is mandatory; default refuses without it)
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --yes

# Dry Run (simulated, no API calls)
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --dry-run

# Auto-rollback when shutdown fails
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --auto-rollback --yes

6-phase execution flow (see references/execution-workflow.md for details):

Phase Action Description
1. Pre-check Query status All instances must be ACTIVE
2. Shutdown BatchStopServers Inject the fault
3. Monitor Poll status Wait until all reach SHUTOFF
4. Wait sleep(duration) Hold with stop-condition checks
5. Rollback BatchStartServers Restore instances
6. Verify Poll status Confirm all back to ACTIVE

Output: execution-log.json with full execution timeline and state changes.

Step 2.2: Generate Execution Report

python3 scripts/generate_report.py --log-file ./experiments/xxx/execution-log.json

Report contains: experiment overview, target instances, phase timeline, state changes, conclusions.

Emergency Rollback

If the experiment must be terminated early:

python3 scripts/rollback_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region>
# Or pass instance IDs directly:
python3 scripts/rollback_experiment.py --ids "i-xxx,i-yyy" --cli-region <region>

Phase 3: Analyze

Collect CES monitoring metrics and LTS application logs to understand how applications respond to the infrastructure failure. Supports post-hoc mode (after experiment) and real-time mode (during experiment).

Step 3.1: Load Experiment Context

Auto-detect mode: execution-log.json found → post-hoc; falls back to experiment.json → real-time.

Step 3.2: Run Full Analysis Pipeline

python3 scripts/analyze_logs.py \
    --experiment-dir ./experiments/xxx/ \
    --cli-region <region> \
    --log-group-id <lts-group-id>

# Dry run (verify workflow without API calls)
python3 scripts/analyze_logs.py --experiment-dir ./experiments/xxx/ --dry-run

The pipeline executes steps 3.3–3.6 automatically. See references/analysis-workflow.md for details.

Step 3.3: Collect CES Monitoring Metrics

Queries 6 metrics via hcloud CES ShowMetricData (see references/managed-service-logs.md → CES section):

Namespace Metric Label Requires Agent?
SYS.ECS cpu_util CPU Utilization No
SYS.ECS network_incoming_bytes_aggregate_rate Network Incoming Rate No
SYS.ECS network_outgoing_bytes_aggregate_rate Network Outgoing Rate No
SYS.ECS disk_read_bytes_rate Disk Read Rate No
SYS.ECS disk_write_bytes_rate Disk Write Rate No
AGT.ECS mem_usedPercent Memory Usage Yes

Metrics summarized by phase (pre-shutdown / during-shutdown / post-recovery). Query uses period=300 (5-min aggregation for SYS.ECS).

Step 3.4: Collect Logs (LTS)

Queries LTS log group within experiment time window. --log-group-id is required (find via hcloud LTS ListLogGroups). Missing group / empty result = no LTS data, NOT "application unaffected" — check ICAgent config.

Step 3.5: Analyze Error Patterns

Local pattern matching (no network calls):

Pattern Severity
ERROR/Exception/Traceback high
Connection refused high
Timeout high
HTTP 5xx high
Reconnect/retry medium
Degrade/circuit breaker medium
Recovery/restored info

Step 3.6: Generate Analysis Report

python3 scripts/generate_analysis_report.py --result-file ./experiments/xxx/log-analysis-result.json

Report contains: experiment overview, affected instances, LTS log sources, instance state timeline, CES monitoring metrics, error pattern statistics, error event timeline, application behavior assessment, improvement suggestions.

Parameter Confirmation

Before each phase's core operation, confirm with the user:

Phase What to Confirm Mandatory?
1 (Generate) Target instance IDs + names, duration, validation warnings ✅ Yes
2 (Execute) Experiment name, target count, duration, dry-run flag ✅ Yes — --yes required for real execution
3 (Analyze) Experiment directory, detected mode, time window, LTS log group ID ✅ Yes

Wait for explicit confirmation ("确认" / "confirm" / "ok") before proceeding.

Core Commands

<region> = region, defaults to cn-north-4 (HW_REGION_NAME overrides if set). The placeholder may be omitted at runtime — scripts fall back to the default region.

# Environment check (all phases)
bash scripts/check_env.sh --cli-region <region>

# Phase 1: Prepare
python3 scripts/discover_ecs.py --cli-region <region>
python3 scripts/validate_targets.py --instances-file instances.json --cli-region <region>
python3 scripts/generate_experiment.py --targets "i-xxx,i-yyy" --cli-region <region> --duration 300
bash scripts/deploy_experiment.sh --experiment-dir ./experiments/{dir}/ --cli-region <region>

# Phase 2: Execute
python3 scripts/execute_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region> --yes
python3 scripts/generate_report.py --log-file ./experiments/xxx/execution-log.json
python3 scripts/rollback_experiment.py --experiment-dir ./experiments/xxx/ --cli-region <region>  # emergency

# Phase 3: Analyze
hcloud LTS ListLogGroups --cli-region=<region> --cli-output=json  # find log group
python3 scripts/analyze_logs.py --experiment-dir ./experiments/xxx/ --cli-region <region> --log-group-id <id>
python3 scripts/generate_analysis_report.py --result-file ./experiments/xxx/log-analysis-result.json

KooCLI Command Format Standard

All hcloud CLI commands follow this format:

hcloud <Service> <Action> --cli-region=<region> --cli-output=json [--param=value ...]
  • --cli-region=<region>: always explicit. KooCLI 7.2.x requires --param=value syntax (equals sign); space-separated --cli-region cn-north-4 is rejected with [USE_ERROR].
  • --cli-output=json: all commands, for machine-parseable output.
  • Batch operations use flat parameters (not --body):
    • Stop: --os-stop.servers.1.id=<id> --os-stop.type=SOFT
    • Start: --os-start.servers.1.id=<id>
  • Fetch in batch, filter locally: ListServersDetails with --limit=1000 + paging; no N+1 queries.
  • Credentials via environment variables: AK/SK passed through HW_ACCESS_KEY / HW_SECRET_KEY, never hardcoded.

Script Reference

Script Phase Purpose
scripts/check_env.sh All Environment verification
scripts/discover_ecs.py 1 Discover ECS instances
scripts/validate_targets.py 1 Validate target compatibility
scripts/generate_experiment.py 1 Generate experiment config
scripts/deploy_experiment.sh 1 Deploy experiment (local mode)
scripts/execute_experiment.py 2 Main execution script (6-phase flow)
scripts/monitor_instances.py 2 Instance status monitoring
scripts/rollback_experiment.py 2 Emergency rollback
scripts/generate_report.py 2 Generate execution report
scripts/collect_logs.py 3 LTS log collection + ECS endpoint query
scripts/collect_ces_metrics.py 3 CES monitoring metrics collection
scripts/analyze_logs.py 3 Main analysis pipeline orchestrator
scripts/generate_analysis_report.py 3 Generate Markdown analysis report

Reference Documents

Document Phase Description
references/cli-installation-guide.md All hcloud KooCLI installation and configuration
references/iam-policies.md All IAM permissions by phase
references/ecs-validation-rules.md 1 Validation rules with rationale
references/experiment-template-guide.md 1 experiment.json schema
references/execution-workflow.md 2 Detailed execution workflow (per-phase I/O, exceptions)
references/monitoring-guide.md 2 Monitoring guide (state machine, polling, CES alarms)
references/analysis-workflow.md 3 Detailed analysis workflow
references/managed-service-logs.md 3 Managed-service log reference (CES, LTS, audit)
references/verification-method.md All Verification methodology (all phases)
references/acceptance-criteria.md All Acceptance criteria (all phases)

Key Safety Features

  1. Dry Run mode — --dry-run simulates the full flow without API calls (Phase 2 & 3)
  2. Auto-rollback — --auto-rollback starts instances back up when shutdown fails
  3. Independent rollback script — rollback_experiment.py is standalone, does not depend on main script state
  4. Full logging — execution-log.json records per-phase timestamps and state changes
  5. Pre-check gate — verifies all instances are ACTIVE before execution
  6. Stop-condition early rollback — CES alarm fires → skip remaining duration → rollback immediately
  7. Mandatory confirmation — real execution requires explicit --yes; otherwise refuses
  8. Environment variable whitelist — child processes receive only necessary vars, not full os.environ

ECS Instance State Machine

Stop:    ACTIVE → STOPPING → SHUTOFF
Start:   SHUTOFF → STARTING → ACTIVE
Abnormal: any → ERROR

Limitations

  • Depends on hcloud CLI with sufficient ECS permissions (BatchStopServers, BatchStartServers, ListServersDetails)
  • AS group check requires AS API access (skipped if unavailable)
  • CES SYS.ECS metrics are aggregated at 5-minute granularity (period=300); shorter periods return empty
  • CES API may be unreachable in sandboxed environments — see references/managed-service-logs.md → CES section
  • AGT.ECS metrics (memory) require telescope/uniagent Agent installed inside the ECS
  • LTS collection requires ICAgent shipping logs to a LTS log group; missing group = no data, not "unaffected"
  • Does not support composite scenarios (e.g., multi-action AZ-level interruption) — focused on single ECS shutdown
  • In real-time mode instance_timeline is unavailable, so recovery-time analysis is skipped
  • Error-pattern analysis is based on string matching and may miss unusual log formats