SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillshuawei-cloud-agentarts-obs-diagnosis

huawei-cloud-agentarts-obs-diagnosis

Agents & models

View repository

Updated 10-15 22:14

About this skill

Huawei Cloud AgentArts Observability No-Data Diagnosis

Overview

This skill systematically diagnoses missing metrics, logs, and traces/sessions in Huawei Cloud AgentArts agent-ops platform. It follows a three-phase diagnostic flow: subscription metadata check → data pipeline troubleshooting → query parameter validation, covering three data sources: APM (traces/metrics), AOM (metrics), and LTS (logs).

Architecture

Application (OTel SDK/Collector)
  │
  ├── Traces:  OTEL SDK → APM (OTLP) → deliverConfig → Kafka → agent-ops
  ├── Metrics: Application → AOM (Prometheus) → agent-ops
  └── Logs:    Application → LTS → agent-ops (requires logGroup/logStream metadata)

Data Sources and Metadata

Data Type Source Required Metadata
Traces/Sessions APM apmBusiness, apmToken
Metrics AOM promInstance, aomAccessCode
Logs LTS logGroup, logStream

Prerequisites

1. Huawei Cloud Credentials

export HUAWEI_ACCESS_KEY="<your-access-key-id>"
read -rs HUAWEI_SECRET_KEY; export HUAWEI_SECRET_KEY
export HUAWEI_REGION="cn-north-4"

2. hcloud CLI (KooCLI)

Install: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html

3. IAM Permissions

See references/iam-policies.md for least-privilege policy.

Workflow

Step 1: Identify Missing Data Type

Use AskUserQuestion to determine which data type is missing:

Option Data Type Data Source
A Metrics (metrics) AOM / APM
B Logs (logs) LTS
C Traces/Sessions (traces/session) APM
D All missing All sources

Record symptoms: complete vs partial absence, error codes, onset time.

Step 2: Confirm Environment Context

Parameter Default Description
region cn-north-4 Query region (must match ingestion region)
domain_id — IAM domain ID
project_id — Project ID for LTS queries
start_time / end_time — Millisecond timestamps (13 digits, NOT 10-digit seconds)

Step 3: Check Subscription Metadata (Phase 1 — Always First)

Missing subscription metadata is the most common root cause of no-data issues.

Check APM business and token:

hcloud APM ListBusiness --cli-region=cn-north-4

If no business exists → APM not subscribed. Create business:

hcloud APM CreateBusiness --cli-region=cn-north-4 \
  --name="obs-diagnosis-check" \
  --display_name="obs-diagnosis-check" \
  --descp="diagnosis" \
  --cmdb_datasource_type=OTEL

Verify business token:

hcloud APM ShowToken --cli-region=cn-north-4 \
  --x-business-id={business_id}

Check AOM Prometheus instance and access code:

hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
hcloud AOM ListAccessCode --cli-region=cn-north-4

If no PromInstance → AOM not configured. Create Prometheus instance (--prom_type must be one of ECS/CCE/REMOTE_WRITE/CLOUD_SERVICE/ACROSS_ACCOUNT; --region header is required and must match --cli-region):

hcloud AOM CreatePromInstance --cli-region=cn-north-4 \
  --prom_name="obs-diagnosis" \
  --prom_type=REMOTE_WRITE \
  --region=cn-north-4

Check LTS log groups and streams:

hcloud LTS ListLogGroups --cli-region=cn-north-4
hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}

If no logGroup/logStream → LTS not subscribed for agent-ops.

Step 4: Diagnose by Data Type (Phase 2 — Pipeline Check)

4A. Metrics No-Data Diagnosis

Check pipeline bottom-up:

  1. Application side: Is OTEL metrics SDK / Prometheus exporter configured?
  2. AOM side: 
    hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0
hcloud AOM ListAccessCode --cli-region=cn-north-4
hcloud AOM ListMetadataAomPromGet --cli-region=cn-north-4
    • Prometheus instance valid?
    • Access code valid?
    • Endpoint reachable?
  3. Query side: Use Prometheus query to verify data exists: 
    hcloud AOM ListInstantQueryAomPromGet --cli-region=cn-north-4 \
  --query="up" \
  --prom_instance_id={prom_id}
  4. agent-ops filter: Check metric_name matches (resource_tokens, model_tokens, session_tokens, trace_tokens), filter labels match (gen_ai_resource_type, resource_id, model_id).

4B. Logs No-Data Diagnosis

  1. Application side: Is ICAgent / OTEL logs exporter configured?
  2. LTS side: 
    hcloud LTS ListLogGroups --cli-region=cn-north-4
hcloud LTS ListLogStreams --cli-region=cn-north-4 --log_group_id={group_id}
hcloud LTS ListLogs --cli-region=cn-north-4 \
  --log_group_id={group_id} \
  --log_stream_id={stream_id} \
  --start_time=1720000000000 \
  --end_time=1720600000000
  3. Query parameters: Check agentRunId, agentId, traceId, logType, keyword syntax (LTS query language).
  4. Error codes:
    • LTS.2446 → Unsupported query syntax — fix keyword syntax
    • 403 → Permission denied — check agency token (ops_admin_trust)

4C. Traces/Sessions No-Data Diagnosis

  1. Application side: 
    hcloud APM SearchAgent --cli-region=cn-north-4 --x-business-id={business_id}
    Is OTEL SDK configured with correct endpoint and token?
  2. APM deliverConfig:
    • Is deliver_otel_trace=true in deliverConfig?
    • Is Kafka reachable (bootstrap-servers, SASL_SSL, topic exists)?
  3. APM data verification: 
    hcloud APM SearchTransaction --cli-region=cn-north-4 \
  --x-business-id={business_id} \
  --start_time=1720000000000 \
  --end_time=1720600000000
  4. agent-ops query side (API only — not in KooCLI): 
    curl -X POST ${AGENT_OPS_ENDPOINT}/v1/ops/observation/traces \
  -H "Authorization: Bearer ${IAM_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"start_time": 1720000000000, "end_time": 1720600000000}'

Step 5: Common Checks (All Data Types)

  1. Timestamp format: Must be 13-digit milliseconds (10-digit seconds → no data)
  2. Region consistency: Ingestion region must match query region
  3. Authentication: IAM token / AppCode valid? 401 → expired, 403 → insufficient permission. ${IAM_TOKEN} Must be replaced at runtime after obtaining via IAM token / AppCode (see the Step 4 agent-ops query command above); if not replaced, it remains a placeholder, and copying the request as-is will definitely return 401.
  4. Permission check: 
    hcloud IAM ListPoliciesV5 --cli-region=cn-north-4

Step 6: Generate Diagnostic Report

See references/diagnostic-report-template.md for the report format.

Core Commands

Command Purpose Mode
hcloud APM ListBusiness --cli-region=cn-north-4 List APM businesses CLI
hcloud APM ShowBusinessDetail --cli-region=cn-north-4 Show business details CLI
hcloud APM ShowToken --cli-region=cn-north-4 Get business token CLI
hcloud APM ShowAccessPoint --cli-region=cn-north-4 Get access point CLI
hcloud APM SearchAgent --cli-region=cn-north-4 Search agents CLI
hcloud APM SearchTransaction --cli-region=cn-north-4 Search transactions (traces) CLI
hcloud APM ShowSpanSearch --cli-region=cn-north-4 Search spans CLI
hcloud APM ListOpenRegion --cli-region=cn-north-4 List open regions CLI
hcloud AOM ListPromInstance --cli-region=cn-north-4 --Enterprise-Project-Id=0 List Prometheus instances CLI
hcloud AOM ListAccessCode --cli-region=cn-north-4 List access codes CLI
hcloud AOM ListInstantQueryAomPromGet --cli-region=cn-north-4 Instant Prometheus query CLI
hcloud AOM ListRangeQueryAomPromGet --cli-region=cn-north-4 Range Prometheus query CLI
hcloud AOM ListMetadataAomPromGet --cli-region=cn-north-4 List Prometheus metadata CLI
hcloud LTS ListLogGroups --cli-region=cn-north-4 List log groups CLI
hcloud LTS ListLogStreams --cli-region=cn-north-4 List log streams CLI
hcloud LTS ListLogs --cli-region=cn-north-4 List logs CLI
hcloud IAM ListPoliciesV5 --cli-region=cn-north-4 List IAM policies CLI
POST {AGENT_OPS_ENDPOINT}/v1/ops/observation/traces Query traces in agent-ops API
POST {AGENT_OPS_ENDPOINT}/v1/ops/observation/metrics/trend Query metric trends API

Parameter Confirmation

Parameter Required Description Example
region Yes Query region cn-north-4
business_id Yes (traces) APM business ID From ListBusiness
prom_instance_id Yes (metrics) AOM Prometheus instance ID From ListPromInstance
log_group_id Yes (logs) LTS log group ID From ListLogGroups
log_stream_id Yes (logs) LTS log stream ID From ListLogStreams
start_time Yes Start time (ms timestamp) 1720000000000
end_time Yes End time (ms timestamp) 1720600000000
metric_name No (metrics) Metric name to query resource_tokens
keywords No (logs) LTS keyword search error AND timeout
trace_id No (traces) Trace ID filter —

KooCLI Command Format Standard

hcloud <Service> <Operation> --cli-region=<region> [--key=value ...]
Feature Description Example
Service name KooCLI Service name (uppercase) APM, AOM, LTS, IAM
Operation name PascalCase ListBusiness, ShowToken
Region parameter --cli-region=<value> --cli-region=cn-north-4
Simple parameter --key=value --business_id=xxx

Reference Documents

  • references/iam-policies.md — Least-privilege IAM policies
  • references/cli-installation-guide.md — hcloud CLI installation guide
  • references/verification-method.md — Verification methods and API status
  • references/dataflow-diagram.md — Mermaid data flow and diagnostic flow
  • references/acceptance-criteria.md — Acceptance criteria for diagnosis
  • references/diagnostic-report-template.md — Diagnostic report template

Edge Cases

Scenario Handling
All data types missing Check subscription status first, then network connectivity
Partial data missing (some apps) Check agent deployment, service.name mapping
LTS.2446 error Fix keyword syntax (escaping, quotes, operators)
403 error (non-claw) Check IAM tenant permissions and subscription status
403 error (claw) Check agency (ops_admin_trust) and STS configuration
RESOURCE_NOT_EXIST Resource records missing — confirm CreateClawTenantRelation
Metadata missing Subscribe first (SubscribeOpsObservation)
Timestamp 10 digits Convert to 13-digit milliseconds
Region mismatch Use same region for ingestion and query

Notes

  • Priority order: Subscription metadata → data pipeline → query parameters → permissions
  • Most common root cause: Missing subscription metadata (apmBusiness, promInstance, logGroup)
  • Timestamp pitfall: 10-digit seconds instead of 13-digit milliseconds
  • Credential security: Never hardcode AK/SK. Read from env vars or CLI profile.
  • Least privilege: See references/iam-policies.md for required permissions.