SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscrawl-xueqiu-my-timeline

crawl-xueqiu-my-timeline

Data & analytics

View repository

Updated 10-15 20:13

About this skill

crawl-xueqiu-my-timeline

Crawl the timeline of followed accounts on the Xueqiu homepage, save it as Markdown files grouped by speaker, for AI to analyze and summarize and generate a PDF investment report.

Features：

  • Automatically filter out official accounts (market updates for listed companies, indices, ETFs, etc.)
  • Output grouped by speaker, sorted in descending order by number of posts
  • Within each group, reverse chronological order, with newer content first

Quick Start

# 1. 检查环境
sh skills/crawl-xueqiu-my-timeline/scripts/check-cdp.sh
sh skills/crawl-xueqiu-my-timeline/scripts/check-agent-browser.sh

# 2. 爬取最近 24 小时时间线
skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py
3. AI automatically reads, analyzes, and generates a PDF report 

## 何时使用

✅ **应该触发**：
- "帮我看看雪球上今天有什么投资要闻"
- "生成一份市场热点分析报告"
- "把关注的雪球大 V 观点整理成 PDF"
- "最近有什么投资热点"
- "爬取雪球时间线"
- "生成投资分析报告"

❌ **不应触发**：
- "打开雪球网站"（只需浏览器操作）
- "查看某只股票行情"（查询单一数据）
- "帮我登录雪球"（简单网页操作）

## 前置准备

确保 Chrome 处于 Debug 模式并安装 agent-browser：

```bash
sh skills/crawl-xueqiu-my-timeline/scripts/check-cdp.sh
sh skills/crawl-xueqiu-my-timeline/scripts/check-agent-browser.sh

AI Task List

  1. ✅ Check environment: Chrome Debug mode + agent-browser
  2. ✅ Run crawl: run crawl_xueqiu_home_timeline_api.py
  3. ✅ Read files: read the generated home_timeline_YYYYMMDD_YYYYMMDD.md
  4. ✅ Create TODO: create an analysis TODO list based on the number of speakers
  5. ✅ Distribute tasks: launch subagents to analyze user posts (concurrency ≤ 3)
  6. ✅ Aggregate results: integrate the analysis results of all subagents
  7. ✅ Generate report: write a Markdown-format investment analysis report
  8. ✅ Convert to PDF: use mdpdf to convert to PDF
  9. ✅ Clean up files: delete temporary Markdown files

AI Analysis Guidelines

⚠️ Important: Determining opinion attribution

Only content that does not start with // or > is the speaker's own opinion：

Marker Meaning Whether to summarize
Unmarked line The speaker's own post ✅ Summarize
// starts with Another user's post being commented on ❌ Not the speaker's own opinion
> starts with Quoted reposted content ❌ Not the speaker's own opinion

Note：回复@XXX： The prefix is automatically generated by Xueqiu; the entire line is the speaker's own opinion and should be summarized.

Example：

回复@阿企笔记：要知道很多农村的人需要方言语音设备
//回复@青年完：支付宝本来就有音响
> @阿企笔记：中概股失去增长动力

Correct summary: The speaker believes rural users need dialect voice devices

Incorrect summary: The speaker discussed Alipay speakers and Chinese concept stock earnings reports

Reading output files

The crawl script will generate Markdown files in the current directory:home_timeline_YYYYMMDD_YYYYMMDD.md

# 读取最新生成的时间线文件
cat home_timeline_*.md

Analysis Report Format

AI should generate an investment analysis report according to the following structure:

# 雪球关注时间线分析报告

## 时间范围
2026-03-05 至 2026-03-06

## 发言人统计
- @产业驱动博弈：19 条
- @管我财：6 条
...（逐一列出所有发言人）

## 发言观点总结

### @发言人 A（发言最多的用户）
- **发言数量**: X 条
- **主要观点**:
  - 观点 1 摘要
  - 观点 2 摘要
- **涉及标的**: $股票 1$, $股票 2$
- **情绪倾向**: 乐观/中性/悲观

### @发言人 B
- **发言数量**: X 条
- **主要观点**:
  - 观点摘要
- **涉及标的**: $股票 3$
- **情绪倾向**: 乐观/中性/悲观

...（继续列出所有发言人，直到最后）...

## 市场热点 TOP3
1. 热点主题 1 - 讨论人数 X
2. 热点主题 2 - 讨论人数 X
3. 热点主题 3 - 讨论人数 X

## 投资建议/风险提示
（基于发言内容的综合分析）

Core Principles：

  • ✅ Must include all speakers, even if there is only 1 post
  • ✅ Sort in descending order by number of posts
  • ✅ Only summarize the person's own opinions (content that does not start with // or > )
  • ✅ Mark speakers whose analysis failed as "⚠️ Analysis failed" in the report

Analysis Dimensions

  1. Group by speaker: aggregate all posts by the same user
  2. Opinion extraction: identify core opinions, deduplicate and merge similar content
  3. Target identification: extract $股票名$ formatted code
  4. Sentiment Analysis: Determine market sentiment based on word choice
  5. Hot Topic Aggregation: Count topics discussed across users

AI Analysis Workflow (Subagent Scheduling)

Step 1: Create TODO List

After reading the timeline file, count each speaker's number of messages, byNumber of messages from a single userDetermine the TODO allocation strategy:

Number of user messages TODO allocation method Description Example
Single user >10 messages One exclusive TODO Users with many messages are analyzed separately @A(15 messages) → TODO: Analyze @A
Single user 5-10 messages 3-5 users merged into one TODO Medium-volume users grouped for analysis @B(7 messages)+@C(6 messages)+@D(5 messages) → TODO: Analyze @B/@C/@D
Single user <5 messages All users merged into one TODO Low-volume users analyzed together @E(3 messages)+@F(2 messages)+@G(1 message) → TODO: Analyze @E/@F/@G

Decision Logic(check in order, stop when satisfied):

  1. Iterate over all speakers and group users by message count:
    • High-output group: message count >10
    • Medium-output group: message count 5-10
    • Low-output group: message count <5
  2. Create TODOs:
    • High-output group: one independent TODO per user
    • Medium-output group: merge every 3-5 users into one TODO (e.g. 10 users split into 2-3 TODOs)
    • Low-output group: merge all users into 1 TODO

Use tools supported by the current environment to manage TODOs (such as OpenCode's todowrite tool)

Step 2: Subagent Instruction Template

Each subagent receives the following instructions:

分析雪球时间线文件中指定用户的发言

**文件路径**: `<时间线文件绝对路径>`

**需分析的用户**：
- @用户 A：19 条发言
- @用户 B：6 条发言

**分析指引**：
1. 读取文件，定位目标用户的发言段落
2. 只总结本人观点（排除 `//` 或 `>` 开头的内容）
3. 提取：核心观点、涉及标的（$xxx$格式）、情绪倾向

**输出要求**：
直接返回分析结果（不要写文件），格式：

### @用户 A
- **发言数量**: 19 条
- **主要观点**:
  - 观点 1
  - 观点 2
- **涉及标的**: $XXX$, $YYY$
- **情绪倾向**: 乐观/中性/悲观

Step 3: Concurrent Scheduling

  • Concurrency limit: no more than 3 subagents running simultaneously
  • Execution workflow：
    1. Fill idle slots (start subagents)
    2. Wait for any subagent to complete
    3. Mark TODO complete/failed
    4. Continue to the next batch

Step 4: Retry Mechanism

  • Automatically retry when a subagent fails, up to 2 times
  • For TODOs that keep failing, mark "⚠️ Analysis failed" in the report and prompt the user that manual analysis is possible

Step 5: Result Aggregation

Integrate the analysis results of all subagents and generate a complete investment analysis report:

  • Merge the analyses of all speakers
  • Identify the TOP3 market hot topics discussed across users
  • Comprehensive investment advice/risk warnings

Environment Adaptation Guide

The main agent should choose an appropriate method based on the tools available in the current environment:

  • OpenCode: use the task tool to start subagents, use todowrite to manage TODOs
  • Claude Code: use a method suitable for the current environment to start subagents
  • Core principle: each subagent's analysis result is returned directly through context (no intermediate files are written)

Crawler Script Parameters

skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py [选项]
Parameter Description Default Value
--hours Crawl the last N hours 24
--days Crawl the last N days -
--start-date Start date (YYYY-MM-DD) -
--end-date End date (YYYY-MM-DD) Today
-o, --output Output file name Automatically generated

Note：--hours、--days、--start-date The three parameters are mutually exclusive.

Example

# 爬取最近 24 小时（默认）
skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py

# 爬取最近 2 小时
skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py --hours 2

# 爬取最近 7 天
skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py --days 7

# 指定日期范围
skills/crawl-xueqiu-my-timeline/scripts/crawl_xueqiu_home_timeline_api.py --start-date 2026-03-01 --end-date 2026-03-06

PDF Output Process

1. Save the analysis report

Save the analysis report as 雪球时间线_YYYYMMDD_YYYYMMDD.md, with the dates being the start and end dates of the time range.

Example:雪球时间线_20260305_20260306.md

2. Convert to PDF

bunx mdpdf 雪球时间线_YYYYMMDD_YYYYMMDD.md \
  --style=/home/cnife/code/my-scripts/skills/crawl-xueqiu-my-timeline/assets/github-markdown.css

For example:

bunx mdpdf 雪球时间线_20260305_20260306.md \
  --style=/home/cnife/code/my-scripts/skills/crawl-xueqiu-my-timeline/assets/github-markdown.css

Optional Parameters：

  • --border=<size>: Page margin (default 20mm), e.g. --border=15mm
  • --format=<format>: Paper size (default A4), e.g. --format=A4
  • --orientation=<orientation>: Orientation (default portrait)
  • --title=<title>: PDF title

3. Clean up temporary files

rm 雪球时间线_YYYYMMDD_YYYYMMDD.md

Complete Example

# AI 写入总结文件 → 雪球时间线_20260305_20260306.md

# 转换为 PDF（当前目录生成 PDF）
bunx mdpdf 雪球时间线_20260305_20260306.md \
  --style=/home/cnife/code/my-scripts/skills/crawl-xueqiu-my-timeline/assets/github-markdown.css

# 删除临时 Markdown 文件
rm 雪球时间线_20260305_20260306.md

Result: Generated in the current working directory 雪球时间线_20260305_20260306.pdf

Output Format

The home_timeline_*.md generated by the crawler script contains:

  • File header: time range, generation time, speech statistics table
  • Grouped by speaker: sorted by number of speeches in descending order
  • Within each group, sorted by time in reverse order: newer speeches come first
  • Each speech: publication time, content, quoted content, original link
  • Automatic filtering: official account market broadcasts

Notes

  1. You need to log in to your Xueqiu account first
  2. If you encounter a Verification page, you need to complete the verification manually and then run again
  3. The crawling process automatically handles pagination and the md5__1038 token
  4. Output files are saved in the current working directory
  5. Subagent Scheduling: Analysis tasks are executed through subagents, with concurrency ≤ 3, and automatic retry twice on failure
  6. Environment Adaptation: The main agent selects the subagent startup method based on the tools available in the current environment

Typical Use Cases

  1. Daily investment news review: Crawl the last 24 hours, and AI summarizes market hotspots and generates a PDF report
  2. Follow key influencer updates: Crawl the last N days, and AI analyzes changes in investment views and generates a PDF report
  3. Track specific events: Specify a date range, and AI organizes discussions during the event and generates a PDF report

Resource Files

  • assets/github-markdown.css: PDF style file (used by mdpdf)
  • scripts/check-cdp.sh: Check Chrome Debug mode
  • scripts/check-agent-browser.sh: Check agent-browser installation
  • scripts/crawl_xueqiu_home_timeline_api.py: Crawler script (main entry point)
  • evals/evals.json: Test case set (used to verify skill functionality)