hiascend-forum-analyzer

Ascend Community Forum Issue Analysis Tool

Feature Overview

This tool is used to analyze posts related to technical issues, troubleshooting, and error handling in the Ascend Community Forum. It supports reading data from standard-format Excel files, automatically identifying issue-type posts, obtaining detailed content, and performing multi-dimensional filtering and analysis.

Input Data Format

Source File Format：hiascend_topics_temp.xlsx

Required Fields：

• topicId: Unique post identifier
• sectionName: Board name
• topicClassName: Topic category
• title: Post title
• createTime: Creation time (format: yyyy-mm-dd)
• topicLink: Post link

File Location: Excel file path specified by the user

Configuration File

Keyword configuration is stored in a separate JSON file for easy maintenance and customization.

Configuration file: references/keywords.json

{
  "initial_filter": {
    "topic_class_keywords": [
      "问题", "故障", "报错", "错误", "error", "issue", "bug", "异常",
      "报错信息", "出错", "失败", "技术支持", "求助", "问答", "提问"
    ],
    "title_keywords": [
      "报错", "错误", "error", "失败", "fail", "异常", "bug", "issue", "问题",
      "无法", "不能", "报错信息", "出错", "运行失败", "编译失败", "安装失败",
      "求助", "请问", "怎么解决", "如何解决", "报错码", "error code"
    ]
  },
  "deep_filter": {
    "problem_keywords": [
      "报错", "错误", "error", "exception", "失败", "fail",
      "无法", "不能", "运行失败", "编译失败", "安装失败", "启动失败",
      "求助", "解决", "报错信息", "报错码", "出错信息",
      "卡死", "死机", "崩溃", "闪退", "无响应",
      "error", "fail", "exception", "bug", "issue", "fault",
      "crash", "cannot", "unable", "not working", "failed"
    ],
    "tech_keywords": [
      "ascend", "cann", "mindspore", "atlas",
      "算子", "模型", "训练", "推理", "部署",
      "驱动", "固件", "sdk", "开发环境"
    ]
  },
  "filter_settings": {
    "case_sensitive": false,
    "match_mode": "contains",
    "min_content_length": 10
  }
}

Configuration Item Description

Configuration Item	Description	Type
initial_filter.topic_class_keywords	topicClassName field filter keywords	String array
initial_filter.title_keywords	title field filter keywords	String array
deep_filter.problem_keywords	In-depth filtering issue-type keywords	String array
deep_filter.tech_keywords	Technical domain keywords (optional)	String array
filter_settings.case_sensitive	Whether case-sensitive	Boolean value
filter_settings.match_mode	Match mode: contains/exact/regex	String
filter_settings.min_content_length	content minimum length limit	Integer

Workflow

1. Read configuration file

• Load references/keywords.json configuration file
• Validate completeness of required configuration items
• Parse filter keywords and matching settings

2. Read and parse Excel data

• Use pandas to read the specified Excel file
• Validate whether required fields exist
• Data type checking and cleaning

3. Preliminary filtering (based on topicClassName and title)

Read filter keywords from the configuration file：

• topicClassName Match keywords: from initial_filter.topic_class_keywords
• title Match keywords: from initial_filter.title_keywords

Matching Settings(from filter_settings）：

• case_sensitive: Whether case-sensitive (default false)
• match_mode: Match mode (contains/exact/regex)
• Supports fuzzy matching, exact matching, and regex matching

Matching Rules：

• Either topicClassName or title matching is sufficient
• Record changes in data volume before and after filtering

4. Obtain detailed post content

API endpoint：

https://www.hiascend.com/ascendgateway/ascendservice/devCenter/bbs/servlet/get-topic-detail?topicId={topicId}

Request Configuration：

• Method: GET
• Timeout: 30 seconds
• Retry count: 3 times
• User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

Response Handling：

• Check HTTP status code (200)
• Parse JSON response
• Extract result.content field
• HTML tag cleaning (optional)

Concurrency control：

• Default concurrency: 5
• Avoid putting pressure on the server
• Add request interval (0.5 seconds)

5. Data structure reorganization

New field：

• content: Detailed post content (placed before topicLink)

Output field order：

topicId, sectionName, topicClassName, title, createTime, content, topicLink

6. Deep filtering (based on title and content)

Read filter conditions from the configuration file：

Problem-type keywords（deep_filter.problem_keywords）：

• 报错, 错误, error, exception, 失败, fail
• 无法, 不能, 运行失败, 编译失败, 安装失败
• 异常, bug, issue, fault, crash, 卡死, 死机
• 求助, 解决, 问题

Technical domain keywords（deep_filter.tech_keywords, optional):

• Ascend、CANN、MindSpore、Atlas
• 算子, 模型, 训练, 推理, 部署
• 驱动, 固件, SDK, 开发环境

Filter mode：

• Mode 1: Contains at least 1 problem-type keyword
• Mode 2: Contains both problem-type and technical domain keywords
• Use the configuration file's filter_settings to control matching behavior

7. Data export

Output file naming：

hiascend_issues_{timestamp}.xlsx

Output fields：

• topicId: Post ID
• sectionName: Board name
• topicClassName: Topic category
• title: Post title
• createTime: Creation time
• content: Post content
• topicLink: Post link

Excel format settings：

• Auto-fit column width
• Enable text wrapping for the content column
• Set the link column to hyperlink format

CLI parameter description

python scripts/analyze_hiascend_issues.py <input_excel> [选项]

Positional arguments：

• input_excel: Input Excel file path (must conform to the hiascend_topics_temp.xlsx format)

Optional arguments：

• --config <路径>: Configuration file path (default references/keywords.json)
• --filter-mode: Filter mode (1=problem-type, 2=problem+technical, default 1)
• --concurrency N: Number of concurrent requests (default 5)
• --output <路径>: Output file path (default auto-generated)
• --log-level <级别>: Log level (DEBUG/INFO/WARNING/ERROR, default INFO)

Dependency requirements

pandas>=1.3.0
openpyxl>=3.0.0
requests>=2.25.0
urllib3>=1.26.0

Installation command:

pip install pandas openpyxl requests urllib3

File structure

hiascend-forum-analyzer/
├── SKILL.md                        # 技能文档
├── scripts/
│   └── analyze_hiascend_issues.py  # 主程序
├── references/
│   └── keywords.json               # 配置文件（关键词和匹配设置）
├── hiascend_topics_temp.xlsx       # 输入文件（示例）
└── hiascend_issues_*.xlsx          # 输出文件（自动生成）

Usage examples

Basic usage (using default configuration)

python scripts/analyze_hiascend_issues.py hiascend_topics_temp.xlsx

Using a custom configuration file

# 创建自定义配置文件 my_keywords.json
{
  "initial_filter": {
    "topic_class_keywords": ["故障", "报错"],
    "title_keywords": ["报错", "错误", "失败"]
  },
  "deep_filter": {
    "problem_keywords": ["报错", "错误", "失败", "无法"],
    "tech_keywords": ["CANN", "MindSpore", "算子"]
  },
  "filter_settings": {
    "case_sensitive": false,
    "match_mode": "contains",
    "min_content_length": 20
  }
}

# 运行分析
python scripts/analyze_hiascend_issues.py hiascend_topics_temp.xlsx \
    --config my_keywords.json \
    --filter-mode 2

Adjusting the filter mode

# 仅筛选问题类帖子
python scripts/analyze_hiascend_issues.py hiascend_topics_temp.xlsx --filter-mode 1

# 同时匹配问题类和技术领域关键词
python scripts/analyze_hiascend_issues.py hiascend_topics_temp.xlsx --filter-mode 2

Adjusting concurrency and output

python scripts/analyze_hiascend_issues.py hiascend_topics_temp.xlsx \
    --concurrency 10 \
    --output ./reports/ascend_issues_march.xlsx

Output result description

Statistics：

• Total input data
• Count after preliminary filtering
• Count of successfully retrieved content
• Final filtered result count

Output file content：

• All fields are consistent with the input file
• Added content column storing detailed post content
• Arranged in original order
• Excel format has been optimized (column width, text wrapping, etc.)

Notes

1. Configuration file: Before first use, ensure references/keywords.json exists and is correctly formatted
2. API access frequency: Default 0.5-second interval to avoid triggering anti-crawling mechanisms
3. Content length: Default limit is 10000 characters; overly long content is automatically truncated
4. Network stability: Supports 3 retries with an exponential backoff strategy
5. Data privacy: Retrieved content is used only for analysis, in compliance with community guidelines
6. Storage space: Data containing content may be large, so ensure sufficient disk space
7. Config hot reload: No need to restart the program after modifying the configuration file; it takes effect automatically on the next run

Error handling

Common errors and solutions：

1. Configuration file error：

   • Error:配置文件格式错误 → Check whether the JSON format is correct
   • Error:配置文件缺少必需项 → Ensure it contains initial_filter and deep_filter

2. Missing Excel fields: Check whether the input file contains all required fields

3. API request failed: Check the network connection and try again later

4. Insufficient memory: Reduce the concurrency level or process data in batches

5. Permission issue: Ensure you have permission to read the input file and write to the output directory

6. Keyword matching issue: Adjust match_mode or the keyword list