Task Objective
This Skill is used to automatically extract documentation content from the official websites of technical frameworks or programming languages, and through intelligent analysis generate standardized Skill sub-skill packages, helping users systematically learn the tech stack.
Core Capabilities
- Intelligent crawling of official website documentation (path filtering based on type)
- Automatic extraction of version numbers and metadata
- LLM content summarization and core capability graph generation
- Templated sub-skill generation (language type/framework type/tool type)
- Automatic validation of skill completeness
Trigger Conditions
Typical user expressions:
- "Help me generate a Vue learning skill package"
- "Create a corresponding skill based on the React official website"
- "I need a Go language skill package"
- "Generate a skill based on the TypeScript official website"
Prerequisites
Dependency Installation
crawl4ai>=0.4.0
playwright==1.40.0
Playwright browser initialization (first use)
playwright install chromium
Operation Steps
1. Information Collection and Preparation
Extract the following information from the user request:
- Official website URL (required)
- Tech stack name (optional, can be inferred from the URL)
- Documentation type (language type/framework type/tool type, optional, can be intelligently inferred)
- Crawl depth (optional, default 2 levels)
2. Website Documentation Crawling (Enhanced Version)
Call scripts/crawl_website.py to crawl the official website content:
python scripts/crawl_website.py \
--url "https://vuejs.org" \
--depth 2 \
--output "/workspace/projects/vue-skills/raw_docs" \
--type framework
Parameter Description:
--url: Official website address--depth: Crawl depth (2-3 levels recommended)--output: Documentation output directory--max-pages: Maximum number of pages to crawl (default 50)--type: Documentation type (language/framework/tool)
Enhanced Features:
- Intelligent path filtering: only crawl documentation paths such as /docs/, /guide/, /api/, etc.
- Automatic version detection: extract version number from the homepage
- Metadata injection: record source URL, crawl time, and version number at the beginning of the file
3. Content Summarization and Analysis
Call scripts/summarize_pages.py to generate a summary report:
python scripts/summarize_pages.py \
--input "/workspace/projects/vue-skills/raw_docs" \
--output "/workspace/projects/vue-skills/summary.json" \
--type framework
Output Content:
- Basic page information statistics
- Documentation structure analysis (API/guide/example distribution)
- Core concept graph (basic version)
- Skill type recommendation
- Priority reading page list
4. Sub-skill Template Selection and Filling
Based on the summary report, the agent selects an appropriate template and fills in the content:
4.1 Select Template
Select from references/templates/ based on the documentation type:
language-skill.md- Programming language type (Go, Python, Rust)framework-skill.md- Frontend framework type (Vue, React, Angular)tool-skill.md- Tool type (Webpack, Docker, ESLint)
4.2 Fill in Metadata
Extract from the summary report:
---
name: <技术栈名称>-skills
description: <能力描述>
metadata:
source_url: <官网URL>
version: <版本号>
crawled_at: <抓取时间>
doc_type: <文档类型>
---
4.3 Fill in Core Content
Based on the summary report and the original documentation, the agent fills in:
- Task Objective: generated based on the core concept graph
- Core Capabilities: extracted from the documentation structure analysis
- Operation Steps: Organize by learning path
- Resource index: Link to the corresponding reference documentation
4.4 Generate reference documentation
Extract key content from the crawled documentation:
core-concepts.md- Detailed explanation of core concepts (>100 lines, including TOC)api-reference.md- API reference documentationbest-practices.md- Best practices guideexamples.md- Practical example code
4.5 Generate code templates
Extract runnable examples from the official documentation:
assets/hello-world.*- Hello World exampleassets/basic-template.*- Basic templateassets/advanced-example.*- Advanced example
5. Skill validation (optional)
Call scripts/validate_skill.py to validate the generated skill:
python scripts/validate_skill.py \
--skill-path "/workspace/projects/vue-skills" \
--output "/workspace/projects/vue-skills/validation_report.json"
Validation content:
- Metadata completeness check
- Directory structure specification check
- Content quality check (description length, line count limits)
- Automatically generate test questions
6. Package and deliver
Call the packaging tool to generate the .skill file:
package_skill(skill_dir_name="<技术栈名称>-skills")
Resource index
Required scripts
-
- Purpose: Crawl official website documentation and convert it to Markdown
- Parameters: url (required), depth, output, type
- Enhanced features: path filtering, version detection, metadata injection
-
- Purpose: Generate content summaries and core capability maps
- Parameters: input (required), output, type
- Output: Summary report in JSON format
-
- Purpose: Validate skill completeness and usability
- Parameters: skill-path (required), output
- Output: Validation report in JSON format
Domain reference
- references/common-doc-patterns.md
- When to read: When analyzing the structure of technical documentation
- Content: Organizational patterns of common technology stack documentation
Skill templates
-
references/templates/language-skill.md
- Applicable scenarios: Programming language skills (Go, Python, Rust)
- Content: Basic syntax, data structures, concurrent programming, Web development
-
references/templates/framework-skill.md
- Applicable scenarios: Frontend framework skills (Vue, React, Angular)
- Content: Core concepts, component development, state management, route configuration
-
references/templates/tool-skill.md
- Applicable scenarios: Tool skills (Webpack, Docker, ESLint)
- Content: Configuration management, plugin system, performance optimization, project integration
Notes
Crawling strategy optimization
- Automatically filter paths based on the documentation type to avoid crawling irrelevant pages
- Control the crawling depth and number of pages to avoid putting pressure on the target website
- Comply with the website's robots.txt rules
Content distillation
- The agent should prioritize reading the summary report to quickly understand the documentation structure
- The core capability map helps design the learning path for sub-skills
- Focus on core documentation such as tutorials, API references, and best practices
Skill templating
- Automatically select the appropriate template based on the documentation type
- Dynamically populate metadata (source URL, version number, crawl time)
- Maintain the consistency and standardization of templates
Validation mechanism
- It is recommended to run the validation script before packaging
- Check whether all validation items pass
- Fix issues based on the validation report
Agent Responsibility Boundaries
- ✅ Agent is responsible for: content analysis, template selection, document generation, example creation
- ✅ Agent is responsible for: designing sub-skill structure based on the summary report
- ❌ Agent is not responsible for: HTML parsing, link discovery, file format conversion
- ❌ Agent is not responsible for: metadata extraction (done by script)
Usage Examples
Example 1: Generate a Vue learning skill (framework type)
# 1. 抓取 Vue 官网
python scripts/crawl_website.py \
--url "https://vuejs.org" \
--depth 2 \
--output "./vue-skills/raw_docs" \
--type framework
# 2. 生成摘要
python scripts/summarize_pages.py \
--input "./vue-skills/raw_docs" \
--output "./vue-skills/summary.json" \
--type framework
# 3. 智能体选择 framework-skill.md 模板并填充内容
# 4. 生成参考文档和代码模板
# 5. 验证技能
python scripts/validate_skill.py \
--skill-path "./vue-skills" \
--output "./vue-skills/validation.json"
# 6. 打包交付
package_skill(skill_dir_name="vue-skills")
Example 2: Generate a Go language skill (language type)
# 1. 抓取 Go 官网
python scripts/crawl_website.py \
--url "https://go.dev/doc" \
--depth 3 \
--max-pages 100 \
--output "./go-skills/raw_docs" \
--type language
# 2. 生成摘要
python scripts/summarize_pages.py \
--input "./go-skills/raw_docs" \
--output "./go-skills/summary.json" \
--type language
# 3. 智能体选择 language-skill.md 模板并填充内容
# 4. 重点关注并发编程、Web 开发等内容
# 5. 打包交付
Example 3: Generate a Webpack skill (tool type)
# 1. 抓取 Webpack 官网
python scripts/crawl_website.py \
--url "https://webpack.js.org" \
--depth 2 \
--output "./webpack-skills/raw_docs" \
--type tool
# 2. 生成摘要
python scripts/summarize_pages.py \
--input "./webpack-skills/raw_docs" \
--output "./webpack-skills/summary.json" \
--type tool
# 3. 智能体选择 tool-skill.md 模板并填充内容
# 4. 重点关注配置管理、插件系统、优化技巧
# 5. 打包交付
Technical Architecture Description
Crawling Strategy Optimization
- Path Filtering: Preset different path filtering rules based on document type (language/framework/tool)
- Version Detection: Automatically extract the version number from the homepage HTML
- Metadata Injection: Inject the source URL, crawl time, and version number into the header of each Markdown file
Content Distillation
- Summary Generation: Use a script to collect basic page information (word count, document type)
- Document Structure Analysis: Identify different types of documents such as APIs, guides, and examples
- Core Capability Graph: Extract potential concepts based on titles and content (basic version)
Skill Templating
- Three Templates: Language type, framework type, and tool type, covering common technology stacks
- Dynamic Filling: Automatically fill in metadata and core content based on the summary report
- Consistency Assurance: Ensure that the generated sub-skills conform to a unified format specification
Verification Mechanism
- Metadata Check: Verify the completeness of the YAML front matter
- Structure Check: Verify that the directory structure conforms to the specification
- Content Quality Check: Verify description length, line count limits, etc.
- Test Question Generation: Automatically generate test questions to verify skill usability