sls-trace-analyzer

SLS Trace Analyzer Skill

Query Alibaba Cloud Log Service (SLS) based on traceId or URL (with timestamp) to retrieve logs, analyze log content, and locate issues in the code.

Usage

/sls-trace <traceId | URL> [--env <environment>] [--region <region>] [--from <date>] [--to <date>]

Parameters

traceId: The traceId to query, or a URL with timestamp (e.g. https://dev.abczs.cn/api-weapp/pe/registration?t=1769398080665）
--env: Optional, environment name: dev/test/prod (default prod, automatically parsed from domain in URL mode)
--region: Optional, region: shanghai/hangzhou (default shanghai, automatically parsed from domain in URL mode)
--from: Optional, start date, format YYYY-MM-DD or YYYY-MM-DD HH:mm:ss (default 3 days ago)
--to: Optional, end date, format YYYY-MM-DD or YYYY-MM-DD HH:mm:ss (default current time)
--longtime: Optional, query only the longtime logstore (default queries both the normal and longtime logstores)

LogStore Query Strategy

Default Behavior: Query both logstores simultaneously to obtain complete logs

Normal storage (prod/test/dev): Regular business logs
Long-term storage (prod_longtime/test_longtime/dev_longtime): Interface request/response details with the @LogReqAndRsp(longTimeLog = true) annotation

Why Query Both：

Normal storage contains MQ messages, gateway logs, etc.
Long-term storage contains detailed request parameters and response data
Querying only a single logstore may miss key information, resulting in incomplete problem analysis

URL Mode Description

When the input is a URL, the skill will automatically:

Parse the environment and region from the domain
Extract the timestamp from the t parameter
First query the gateway access-log to obtain the X-B3-TraceId
Then use the traceId to query detailed logs

Configuration

Python runtime environment (automatically initialized as needed)

sls-query.py、requirements.txt、.venv/ are all located inthe same directory as this SKILL.md。

Invocation Rules：

Directly use the .venv/bin/python in the same directory to execute sls-query.py, no prior check needed
Only when execution fails (No such file or directory、ModuleNotFoundError), execute python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt in the same directory to initialize and retry
Do not use the system python3 to directly run sls-query.py, otherwise dependencies will be missing

Shanghai Region (default)

Endpoint: cn-shanghai.log.aliyuncs.com
Project: abc-cis-log
Credentials: loaded from ~/.config/sls-query/credentials.json

Hangzhou Region

Endpoint: cn-hangzhou.log.aliyuncs.com
Project: abc-cis-log-hangzhou
Credentials: loaded from ~/.config/sls-query/credentials.json

LogStore Naming Rules

Development environment: dev or dev_longtime
Test environment: test or test_longtime
Production environment: prod or prod_longtime

Domain to Environment Mapping

Domain	Environment	Region
`dev.abczs.cn`	dev	shanghai
`test.abczs.cn`	test	shanghai
`abcyun.cn`	prod	shanghai
`region2.abcyun.cn`	prod	hangzhou

Instructions

When the user invokes this skill, execute the following steps:

Step 0: Determine the input type (URL or TraceId)

First determine whether the user input is a URL or a direct traceId:

if input.startswith('http://') or input.startswith('https://'):
    # URL 模式 - 执行 Step 0.1 ~ 0.3
    pass
else:
    # TraceId 模式 - 跳到 Step 1
    trace_id = input

Step 0.1: Parse the URL

from urllib.parse import urlparse, parse_qs

parsed = urlparse(url)
domain = parsed.netloc  # e.g., "dev.abczs.cn"
path = parsed.path      # e.g., "/api-weapp/pe/registration"
query = parsed.query    # e.g., "t=1769398080665" 或 "1769398080665"

# 提取时间戳（支持两种格式）
if '=' in query:
    params = parse_qs(query)
    timestamp = params.get('t', [None])[0]
else:
    timestamp = query  # 直接是时间戳

Step 0.2: Parse the environment and region from the domain

DOMAIN_MAPPING = {
    "dev.abczs.cn": {"env": "dev", "region": "shanghai"},
    "test.abczs.cn": {"env": "test", "region": "shanghai"},
    "abcyun.cn": {"env": "prod", "region": "shanghai"},
    "region2.abcyun.cn": {"env": "prod", "region": "hangzhou"},
}
config = DOMAIN_MAPPING.get(domain, {"env": "prod", "region": "shanghai"})

Step 0.3: Query the Gateway Access Log to obtain the TraceId

Use the timestamp and path to query the gateway log, extract X-B3-TraceId：

.venv/bin/python sls-query.py \
  --mode access-log \
  --timestamp "<timestamp>" \
  --path "<path>" \
  --region "<region>" \
  --env "<env>"

[Important] Gateway Log Topic Description

abc-cis-gateway-service: Business request API gateway entry, used to query access-log to obtain the TraceId
abc-invoke-gateway-service: Cross-region call gateway entry, used to determine whether a cross-region query is needed

[Query Tips]

Time range: By default, query 10 minutesbefore and after the timestamp (not 30 seconds, which is too narrow and easily misses entries)
Path matching: Use the path'skeywordsrather than the full path, for example use registration instead of /api-weapp/pe/registration

If the access-log mode cannot find it, you can query directly using the gateway topic:

# 备用查询方式：直接查 gateway 日志
query = '__topic__: abc-cis-gateway-service and <path关键词>'

Extract the X-B3-TraceId field from the returned logs, then continue with Step 1.

Step 1: Parse parameters and determine configuration

Parse the traceId and optional parameters from the user input, and determine:

# 根据 region 选择配置
if region == "hangzhou":
    endpoint = "cn-hangzhou.log.aliyuncs.com"
    project = "abc-cis-log-hangzhou"
else:  # 默认上海
    endpoint = "cn-shanghai.log.aliyuncs.com"
    project = "abc-cis-log"

# 根据 env 和 longtime 确定 logstore
logstore = env  # dev/test/prod
if longtime:
    logstore = f"{env}_longtime"

# Credentials loaded from ~/.config/sls-query/credentials.json
# (or SLS_ACCESS_KEY_ID / SLS_ACCESS_KEY_SECRET env vars)

# 时间范围（默认最近 3 天）
from_time = from_date or (now - 3 days)
to_time = to_date or now

Step 2: Query SLS logs

Query logs using a Python script:

.venv/bin/python sls-query.py \
  --trace-id "<traceId>" \
  --region "<region>" \
  --env "<env>" \
  --from "<from_time>" \
  --to "<to_time>"

If using the longtime logstore, add the --longtime parameter.

Step 2.5: Cross-Region Log Tracing

The first query uses the Shanghai region by default; determine whether other regions need to be queried.

[Decision Logic]

First check whether the Shanghai logs already contain complete error information
- If there are already ERROR logs and exception stack traces for the target service (e.g., abc-his-xxx-service) → No cross-region query needed
- If there are only logs for abc-cis-monitor-service or abc-cis-gateway-service → continue to the next step

Check the region field to determine the actual request routing

his-region1 / region1 → 上海 (已查询，无需再查)
his-region2 / region2 → 杭州 (需要查询)

Conditions requiring a cross-Region query(must all be satisfied)
- Logs show region: his-region2 or region2
- In the Shanghai logs,missingthe target service's detailed error stack trace

[Execute Cross-Region Query]

When the above conditions are met, immediately execute the Hangzhou query (no need to ask the user):

.venv/bin/python sls-query.py \
  --trace-id "<traceId>" \
  --region "hangzhou" \
  --env "<env>"

[Architecture Description]

abc-cis-monitor-service is deployed globally only in Shanghai, recording HTTP request summaries for all regions
Shanghai logs: monitor layer (request/response summaries)
Hangzhou logs: detailed business service logs, ERROR stack traces

Step 3: Analyze log content

After obtaining the logs, perform the following analysis:

Identify errors and exceptions
- Look for ERROR and WARN level logs
- Identify Java exception stack traces (Exception, Caused by)
- Extract error messages and error codes
Trace the request chain
- Sort log entries by time
- Identify request entry points (Controller) and exit points
- Mark RPC/Feign calls and responses
- Identify database operations
Extract key information
- Request URL and method
- Request parameters
- Response result
- Time consumption information
- Service call chain

Step 4: Locate the code issue

Based on the log analysis results:

Locate the service from the URL Path
- Extract the request URL path from the logs (e.g., /api/v2/healthlink/xxx）
- View the Gateway route configuration in the Config project:/Users/wxd/IdeaProjects/AbcCisConfig
- Example route configuration format:
```
routes:
  - id: abc-cis-healthlink-service
    uri: http://abc-cis-healthlink-service
    predicates:
      - Path=/api/v2/healthlink/**
```
- id field is the service name
- Find the corresponding service code in the ~/IdeaProjects/ directory based on the service name
- For example:/api/v2/healthlink/** corresponds to service abc-cis-healthlink-service, and the code is located in ~/IdeaProjects/AbcCisHealthlinkService
[Important] Ensure the code branch is consistent with the online environment

2.1 Determine the target environment

Extract environment information from multiple sources in the following priority order:

Source A: The description text entered by the user
- Format example:【重要】[region1][灰度环境][primary]异常请求
- Keyword mapping:
  
  Keyword Environment
  
  灰度环境 gray
  
  预发布环境 pre
  
  正式环境 prod
  
  测试环境 test
  
  开发环境 dev
Source B: The env field of the Gateway access-log
- From __topic__: abc-cis-gateway-service 's logs, extract the env field
- Example:env: gray、env: pre、env: prod、env: dev、env: test
Source C: The logstore environment used by the SLS query
- If none of the above sources are available, infer based on the --env parameter determined in Step 1
2.2 Environment to branch mapping

Environment (env) Git branch

dev dev-joint

test test-joint

pre rc

gray gray

prod master

2.3 Branch usage strategy (by scenario)

Scenario A: The current working directory is already inside the target service project
- Prefer to use the current branch, and directly search for code and locate the issue on the current branch
- Only when on the current branchthe corresponding class/method cannot be found, orthe code line numbers do not match the stack trace, switch to the target branch corresponding to the environment
Scenario B: The current working directory is not inside the target service project
- After entering the target service code directory, directly switch to the target branch corresponding to the environment
```
# 场景 B 或场景 A 回退时的切换流程：
cd ~/IdeaProjects/<ServiceProject>
git fetch origin <target_branch>
git checkout <target_branch>
git pull origin <target_branch>
```
[Notes]
- Before switching branches, first check whether the working tree is clean (git status); if there are uncommitted changes, you need to first git stash
- If the target branch does not exist, fall back to using the master branch
- For dependency libraries (abc-cis-core, abc-cis-commons, abc-bis-rpc-sdk)no branch switch is needed, just git pull on the current branch to pull the latest code
Locate from the exception stack trace
- Extract the class names and line numbers from the stack trace
- Focus first on cn.abcyun code under the package
- Use codebase-retrieval or the Grep tool to find the corresponding code
- Analyze the code logic to find the possible cause of the issue
Locate from the error message
- Search for the location of the error message in the code
- Analyze the conditions under which the error is thrown
- Trace the call chain
Check the relevant dependency code
- If abc-cis-core is involved, check /Users/wxd/IdeaProjects/AbcCisCore
- If abc-cis-commons is involved, check /Users/wxd/IdeaProjects/AbcCisCommons
- If abc-bis-rpc-sdk is involved, check /Users/wxd/IdeaProjects/AbcBisRpcSDK

Keyword	Environment
`灰度环境`	gray
`预发布环境`	pre
`正式环境`	prod
`测试环境`	test
`开发环境`	dev

Environment (env)	Git branch
dev	`dev-joint`
test	`test-joint`
pre	`rc`
gray	`gray`
prod	`master`

[Important] Downstream service code analysis

When the logs show that the error comes from a downstream service called via RPC/Feign,you mustcheck the specific implementation of the downstream service:

Identify downstream service: Identify the target service from the FeignClient class name

FeignClient class name	Corresponding service	Code location
`AbcCisCrmFeignClient`	CRM service	`~/IdeaProjects/AbcCisCrmService`
`AbcCisGoodsFeignClient`	ScGoods service	`~/IdeaProjects/AbcCisScGoodsService`
`AbcCis{Xxx}FeignClient`	Corresponding Xxx service	`~/IdeaProjects/AbcCis{Xxx}Service`

Locate the interface implementation:
1. Find the method name called by FeignClient from the stack (e.g., tryLockingGoodsStockBatch）
2. Find the FeignClient interface definition in the RPC SDK:~/IdeaProjects/AbcBisRpcSDK
3. Based on @PostMapping/@GetMapping annotation, find the URL path
4. Search for the corresponding Controller and Service implementation in the downstream service
Analyze the implementation logic:
1. Read the Controller layer code to understand input parameter handling
2. Read the Service layer code to understand the business logic
3. Pay special attention to: transaction boundaries, locking mechanisms, database operation order
4. For deadlock issues: analyze the SQL execution order and lock acquisition order

Step 4.5: Comprehensive analysis of upstream and downstream logs (deep root cause localization)

[Key] Looking only at the exception stack is not enough to locate the root cause; you must combine upstream and downstream logs and code logic for comprehensive analysis:

Identify all services in the call chain

From the log's __topic__ field, identify the services involved

Common service mappings:

topic	Service name	Code location
`abc-cis-dispensing-service`	Medication dispensing service	`~/IdeaProjects/AbcCisDispensingService`
`abc-cis-sc-goods-service`	Commodity inventory service	`~/IdeaProjects/AbcCisScGoodsService`
`abc-cis-charge-service`	Billing service	`~/IdeaProjects/AbcCisChargeService`
`abc-cis-crm-service`	CRM service	`~/IdeaProjects/AbcCisCrmService`

Analyze logs grouped by service
- Group the logs of the same TraceId by __topic__ field
- Analyze the internal flow of each service separately
- Focus on ERROR level logs and exception stacks
Analyze the upstream service (caller)
- Check whether the call parameters are correct
- Check the call timing and concurrency situation
- Analyze whether the retry logic is reasonable
Analyze the downstream service (callee)
- You must read the code implementation of the downstream service
- Check the interface input parameter handling logic
- Analyze the business processing flow
- Check how transactions and locks are used
Deep analysis of specific problem types

Deadlock problem (Deadlock):
- Check the SQL execution order of the downstream service
- Analyze whether there is inconsistent lock ordering
- Check whether the transaction scope is too large
- Check whether there are concurrent requests operating on the same data
Timeout problem (Timeout):
- Analyze the time consumed at each stage
- Check whether there are slow SQL queries
- Check whether there are external dependencies blocking
Null pointer problem (NullPointer):
- Locate the specific code line
- Analyze the data source
- Check whether the upstream parameters are complete
Comprehensive analysis output
- Explain why this error occurs in conjunction with the code logic
- Provide the specific code fix location and solution

Step 5: Output the analysis report

Generate a structured analysis report:

## 日志分析报告

### 基本信息
- TraceId: xxx
- 环境: prod/test/dev
- 地域: shanghai/hangzhou
- 时间范围: xxx ~ xxx
- 日志条目数: xxx
- 涉及服务: xxx, xxx

### 错误摘要
- 错误类型: xxx
- 错误消息: xxx
- 发生时间: xxx
- 发生服务: xxx

Call chain

1. [入口] POST /api/xxx → DispensingService
2. [Service] DispensingService.dispense()
3. [RPC] → ScGoodsService.lockingGoodsStock()
4. [DB] SQL 执行
5. [Error] Deadlock / NullPointer / Timeout

Upstream and downstream log analysis

#### 上游服务 (xxx-service)
- 调用参数: ...
- 调用时间: ...
- 关键日志: ...

#### 下游服务 (xxx-service)
- 接收参数: ...
- 处理流程: ...
- 错误日志: ...
- 异常堆栈: ...

Root cause analysis (key focus)

#### 问题现象
简述观察到的错误现象

#### 代码分析
- **上游调用代码**: `文件路径:行号`
  ```java
  // 关键代码片段

Downstream implementation code: 文件路径:行号
```
// 关键代码片段
```

Root cause

In conjunction with the code logic, explain why this error occurs:

Fix recommendations

Specific fix solution
Code modification location


## Examples

```bash
# 查询生产环境上海 region 的日志（默认最近 3 天）
/sls-trace abc123def456

# 查询测试环境的日志
/sls-trace abc123def456 --env test

# 查询杭州 region 生产环境的日志
/sls-trace abc123def456 --region hangzhou

# 指定日期范围查询
/sls-trace abc123def456 --from 2025-01-20 --to 2025-01-24

# 指定精确时间范围
/sls-trace abc123def456 --from "2025-01-20 10:00:00" --to "2025-01-20 12:00:00"

# 查询 longtime logstore（历史日志）
/sls-trace abc123def456 --env prod --longtime

# 组合使用
/sls-trace abc123def456 --env test --region hangzhou --from 2025-01-20

# ========== URL 模式示例 ==========

# 通过 URL 自动查询（带 t= 参数）
/sls-trace https://dev.abczs.cn/api-weapp/pe/registration?t=1769398080665
# 自动解析: env=dev, region=shanghai, timestamp=1769398080665

# 通过 URL 自动查询（时间戳直接作为参数）
/sls-trace https://test.abczs.cn/api/v2/charge/sheets?1769398080665
# 自动解析: env=test, region=shanghai, timestamp=1769398080665

# 生产环境 URL
/sls-trace https://abcyun.cn/api/v2/crm/patients?t=1769398080665
# 自动解析: env=prod, region=shanghai

# 杭州 region 生产环境 URL
/sls-trace https://region2.abcyun.cn/api/v2/his/emr?t=1769398080665
# 自动解析: env=prod, region=hangzhou