generate-sls-skill

Generate SLS Skill

Path variables：<project_dir> project-level directory (e.g. .input/my-project/） · <logstore_dir> = <project_dir>/<logstore_name>/

Two input methods are supported:

SLS Project name: automatically via aliyun CLI to fetch index/dashboard data, then generate the SKILL knowledge document
Local directory(input directory): generate the SKILL knowledge document directly from an existing data directory

Input directory structure(either fetch script output or manually prepared):

<logstore_dir>/
├── index.json              # 必需：logstore 索引配置
├── dashboards/             # 资源目录：关联的仪表盘 JSON
│   └── *.json
├── alerts/                 # 资源目录：关联的告警规则 JSON
│   └── *.json
├── scheduled_sqls/         # 资源目录：关联的定时 SQL JSON
│   └── *.json
└── saved_searches/         # 资源目录：关联的快速查询 JSON
    └── *.json

A valid directory must contain index.json(generated by fetch_sls_data.py ).

Execution conventions

⚠️ Mustbe executed in the workspace root directory; cd into the skill directory is prohibited. Violating this convention will cause relative path resolution to fail.

Prerequisites

When the input is an SLS project name, aliyun CLI >= v3.0.308 must be installed and authentication configured (thealiyun sls subcommand was introduced in v3.0.308)
- Installation reference:https://github.com/aliyun/aliyun-cli
- Version verification: run aliyun version, confirm the version number >= 3.0.308
- Authentication verification: run aliyun configure list, confirm a profile with AccessKey configured exists; if there is no profile, prompt the user to first run aliyun configure to configure AccessKey ID/Secret and Region
When executing fetch or validate_queries, it must be run in anon-sandbox environment(the sandbox restricts access to the system certificate chain, causing TLS failures)

Workflow

Checklist(check off during execution):

Step 0: Resume detection (--resume-check -> route by action)

Phase A(project-level preparation, executed once globally):

Steps 1-3 are executed once for the entire project, processing all logstores at once.

Step 1: Locate data source (SLS fetch / local / keyword search)
Step 2：prepare_project.py batch preprocessing
Step 3: Batch filtering + confirm output path <- global checkpoint, do not enter Phase B before completion

Phase B(per-logstore generation):

For each logstore (<logstore_dir>) execute:

update_status.py <project_dir> --mark-in-progress <logstore>
Step 4: Reference document extraction (conditional execution)
Step 5: Generate field descriptions
Step 6: Curated queries
Step 7: Normalized templates
Step 8: Validate queries (optional)
Step 9: Cleanup + annotation
Step 10: Render fragments + report
Step 11: Assemble logstore reference document
update_status.py <project_dir> --mark-completed <logstore>

Phase C(project skill entry update):

Step 12: Generate/update project SKILL.md

Phase D(audit, optional):

Step 13: Audit

Data file reference

<project_dir>/project_summary.json: Step 2 output -> Step 3 input. <project_dir>/selected_logstores.json: Step 3 script output -> Phase B / Step 12 input.

<logstore_dir>/ Working files under

<logstore_dir>/
  ├── skill_options.json                 # Step 1 写入，多步骤读取
  ├── parsed/                            # 中间数据
  │   ├── fields.json                    # Step 2：原始字段（保留）
  │   ├── queries.json                   # Step 2：已去重（不可变，id: q0, q1...）
  │   ├── prepare_summary.json           # Step 2：统计
  │   ├── reference_queries.json         # Step 4（条件，id: r0, r1...）
  │   ├── field_annotations.json         # Step 5 LLM
  │   ├── query_selection.json           # Step 6 LLM
  │   ├── query_pipeline.json            # Step 6 脚本 → Step 7 归一化(+pre_cleaned_query) → Step 8c 更新
  │   ├── query_validation.json           # Step 8a 脚本 → Step 8b 写回 pass/error
  │   ├── query_validation_LLM.json      # Step 8a 脚本（条件） → LLM 处理 → Step 8b 写回
  │   ├── query_annotations.json         # Step 9 LLM
  │   └── query_report.md                # Step 10 脚本
  └── fragments/                         # 组装片段
      ├── datasource.md                  # Step 2（不可变）
      ├── fields_table.md                # Step 5 脚本渲染（render_fields.py）
      ├── queries_selected.md            # Step 10（selected 或 extra 非空时）
      ├── queries_extra.md               # Step 10（条件）
      └── common_values.md               # Step 10 脚本（合并 reference 查询值，空时不生成）

queries.json Each query in source_type contains the"dashboard"、"alert"、"scheduled_sql"、"saved_search") and logstore field, used for reference during curation, classification, and Step 3 inference.

Step 0: Recovery Detection

LLM: Infer from user input <project_dir>(SLS project name → .input/<project>/; local path → use directly or take the parent directory; keyword → search the workspace). If it cannot be determined or the directory does not exist, skip directly to Step 1.

Script: Run recovery detection:

python3 scripts/update_status.py <project_dir> --resume-check

Output example:

{
  "action": "resume_phase_b",
  "summary": {
    "total": 100,
    "pending": 68,
    "in_progress": 2,
    "completed": 27,
    "failed": 3
  },
  "pending_logstores": ["logstore_c", "logstore_d", "..."],
  "in_progress_logstores": ["logstore_a", "logstore_b"],
  "failed_logstores": [
    {"name": "logstore_x", "failed_step": 6}
  ]
}

Based on the action field, jump to the corresponding phase,action meaning:

"first_run" -- no selected_logstores.json -> execute Step 1 normally
"resume_phase_b" -- has pending/in_progress -> jump to Phase B
"all_completed" -- all completed -> jump to Phase C (Step 12)

LLM: Report the progress in the summary to the user, and route to the corresponding step according to action.

Step 1: Locate Data Source

The user input comes in four forms, matched by priority:

SLS Project name: The user provides a project (optionally with a logstore name) → continue with this step
Local path (project directory): Verify that a logstore subdirectory exists under the directory → jump to Step 2
Local path (logstore directory): Verify the directory structure (containing index.json and at least one resource subdirectory) → infer project_dir = parent directory, logstore = directory name → use at Step 2 --logstores=<logstore_name>
Keyword: Search the workspace for a matching directory (containing index.json) → after finding it, jump to Step 2

The following is executed only for SLS Project input.

Confirm that aliyun CLI is available:

Run aliyun version, confirm the version >= 3.0.308; if the version is too low or not installed, prompt the user to upgrade/install
Run aliyun configure list, confirm that a profile with configured AccessKey exists; if there is no profile, prompt the user to first run aliyun configure to configure AccessKey ID/Secret and Region

Determine parameters：

Project only: Do not pass --logstores, fetch automatically handles all logstores with associated resources (internal logstores are automatically filtered out)
project + logstore(s): Pass --logstores=<ls1,ls2,...>, only fetch the logstore specified by the user

Execute fetch(must be run in a non-sandbox environment):

python3 scripts/fetch_sls_data.py <project> .input/<project>/ \
  [--logstores=<ls1,ls2,...>] \
  > /tmp/sls_fetch_summary_<project>.json \
  2>/tmp/sls_fetch_err_<project>.log

For parameters and output format, see the docstring at the top of fetch_sls_data.py .

Result check: Read /tmp/sls_fetch_summary_<project>.json, check whether errors is empty. If not empty, check the specific errors (common ones: API authentication failure, dashboard does not exist).

Data summary: Display the data_summary.md content in stdout, and report the data filtering flow to the user.

Post-fetch processing: Based on the logstores_processed in the summary, display the number of associated resources for each logstore. Check whether there is a "No index config" prompt in warnings —these logstores will be automatically skipped at Step 2 (no index.json）。

User option persistence(parse the user instruction, call the script to batch write skill_options.json）：

Parse optional parameters in the user instruction:

validate_queries: The user mentions "validate query", "check query", etc. → add --validate-queries
reference_source：
- User-specified reference directory: add --reference-dir <path>(script auto-matches by filename)
- User explicitly specifies logstore + document: add --reference <logstore>=<file>

python3 scripts/save_options.py <project_dir> \
  [--validate-queries] \
  [--reference-dir <path>] \
  [--reference <logstore>=<file> ...]

The script writes for each logstore skill_options.json, ensuring subsequent steps can independently read user options.

Note：save_options.py does not need the --logstores parameter—it automatically identifies valid logstores by checking index.json (i.e., logstores already processed in the Step 1 fetch stage).

Step 2: Batch preprocessing

Process all valid logstores under the entire project in a single call:

python3 scripts/prepare_project.py <project_dir> \
  > /tmp/sls_prepare_summary_<project>.json 2>/tmp/sls_prepare_err_<project>.log

Automatically process <project_dir> all valid logstores under it (pure local parsing, executes very quickly). For parameters and output format, see the docstring at the top of prepare_project.py .

Result check: read <project_dir>/project_summary.json, check whether errors is empty.

Step 3: Name simplification + confirm output paths

Global checkpoint (Phase A wrap-up); do not enter Phase B until processing is complete.

Perform name simplification and deduplication for the project and all logstores, and confirm output paths. Read naming_rules.md, execute strictly according to the rules.

Input:<project_dir>/project_summary.json(Step 2 output) Output:<project_dir>/selected_logstores.json + each confirmed logstore's skill_options.json append output_path（<output_root>/<project_alias>/references/<logstore_alias>.md）

Phase B begins

Run python3 scripts/update_status.py <project_dir> --resume-check to get the current status:

pending_logstores: not started → start a new task

in_progress_logstores: in progress (possibly another agent is executing, or it was interrupted last time) → do not start again

failed_logstores: failed → manual intervention required

Execution mode(choose according to platform capabilities):

Parallel(platform supports multiple tasks/sub-agents): start an independent task for each logstore, each executing Steps 4-11

Rate-limit protection: the number of simultaneously running agents must not exceed 8

Serial(single logstore or platform does not support parallelism): execute Steps 4-11 logstore by logstore

Prohibitedcross-logstore interleaved execution (e.g., first do all Step 5 then all Step 6), to avoid context pollution.

Progress monitoring: periodically run --resume-check to refresh the status.

End condition: when pending_logstores and in_progress_logstores are both empty, Phase B is complete; enter Phase C.

The following Steps 4-11 are executed in the context of the current logstore (<logstore_dir>).

Step-level checkpoint: mark progress after each step is completed, supporting resumption from the last completed position after an interruption.

Detect recovery point：python3 scripts/update_status.py <project_dir> --step-resume-check <logstore>

Output example:{"resume_from": 8}

Jump to the corresponding step according to resume_from ; if it is 4, start from the beginning; if it is 12, it is already complete.

Mark in progress：python3 scripts/update_status.py <project_dir> --mark-in-progress <logstore>

Step 4: Reference document extraction

Conditional step: execute only when skill_options.json contains the reference_source field; otherwise skip.

Read reference_extract.md, execute strictly according to the rules.

Input:skill_options.json in reference_source the specified reference document Output:parsed/reference_queries.json(same schema as queries.json , id prefix r）

Step 5: Generate field descriptions

Generate a short Chinese description for each field. Read field_desc.md, and execute strictly according to the rules.

Input:parsed/fields.json(field list generated in Step 2) Output:fragments/fields_table.md

Step 6: Curate queries

The LLM handles semantic selection, and the script handles data assembly.

Read query_select.md, and execute strictly according to the rules.

Input:parsed/queries.json（Step 2） + parsed/reference_queries.json(Step 4, if present) Output:parsed/query_pipeline.json(complete pipeline, the data source for subsequent steps)

Step 7: Normalize templates

Run the normalization script to unify all template syntax into the <var;default> / <var> format, and generate pre_cleaned_query(strip ;default）：

python3 scripts/normalize_templates.py <logstore_dir>

Pure script execution, no LLM required.

Input:parsed/query_pipeline.json(Step 6 output) Output:parsed/query_pipeline.json(updated: each query gains normalized_query and pre_cleaned_query）

pre_cleaned_query = normalized_query in <var;default> is automatically stripped to <var>, for review by the Step 9 LLM.

Record step success：python3 scripts/update_status.py <project_dir> --mark-step <logstore> --step 7

Step 8: Validate queries (optional)

Execute only when skill_options.json in validate_queries is true; otherwise skip.

Perform syntax validation on the queries in the pipeline, remove failed items and backfill. Read query_verify.md, and execute strictly according to the rules.

Input:parsed/query_pipeline.json(Step 7 output, including normalized_query) Output:parsed/query_pipeline.json(updated: remove failed items, backfill, add validation statistics)

Step 9: Cleanup + annotation

Must be handled directly by the LLM; writing a script as a substitute is not allowed.

Read query_format.md, and execute strictly according to the rules.

Input:parsed/query_pipeline.json(Step 7/8 output) Output:parsed/query_annotations.json(standalone annotation file, containing only id/title/category/cleaned_query)

Step 10: Render snippets + report

Run the rendering script:

python3 scripts/render_queries.py <logstore_dir>

The script reads parsed/query_pipeline.json + parsed/query_annotations.json, and after merging generates:

fragments/queries_selected.md(when selected or extra is non-empty)
fragments/queries_extra.md(when extra is non-empty)
parsed/query_report.md(required)

The script also reads parsed/prepare_summary.json、parsed/reference_queries.json(if present) and parsed/query_validation.json + parsed/query_validation_LLM.json(if present) for the report. This step is pure script execution, no LLM required.

Record step success：python3 scripts/update_status.py <project_dir> --mark-step <logstore> --step 10

Step 11: Assemble the logstore reference document

Referring to <logstore_dir>/fragments/queries_selected.md(query categories) and fragments/fields_table.md(field composition), generate:

name: a short readable name (semantic translation preferred, e.g. xxx_audit_log → "Audit Log (xxx_audit_log)"; use the original name when translation is not possible)
description: a one-sentence functional positioning; infer the system and purpose from the query categories and fields; directly listing category names or applying generic templates is prohibited

Run the assembly script:

python3 scripts/assemble_overview.py <logstore_dir> \
  --name "..." --description "..."

The assembly result is written to the reference document path specified by skill_options.json in output_path ; if extra queries exist, also write to the same directory <logstore_alias>-queries-extra.md。

Record step success：python3 scripts/update_status.py <project_dir> --mark-step <logstore> --step 11

Record processing complete：python3 scripts/update_status.py <project_dir> --mark-completed <logstore>

Phase B ends: return to process the next pending logstore, or proceed to Phase C.

Step 12: Generate/update the project SKILL entry

Global one-time execution (Phase C).

Generate or update based on all reference documents <output_root>/<project_alias>/SKILL.md。Read index_rules.md, execute strictly according to the rules.

Input:<project_dir>/selected_logstores.json(including the output_path and output_root of all confirmed logstores) Output:<output_root>/<project_alias>/SKILL.md

Phase C complete: notify the user that SKILL knowledge document generation is complete, and ask whether a quality audit is needed (Phase D, optional, recommended to be executed in a new session).

Phase D starts(optional, recommended in a new session): execute only when the user explicitly requests an audit or the user confirms after Phase C completion that a quality review is needed. Auditing is strongly recommended to be performed in an independent session to obtain an objective perspective and avoid contextual bias from the generation phase.

Step 13: Audit

Optional step, used to audit the semantic quality of LLM-generated content.

Read audit_rules.md, execute strictly according to the rules.

Input:<project_dir>/selected_logstores.json(list of completed logstores) Output:<project_dir>/_audit/(audit plan, per-logstore results, summary report)

Troubleshooting

See troubleshooting.md。