content-source-aggregator

Information Source Hotspot Collector

Freely collect trending content from 6 major platforms and output a standardized JSON hotspot pool.

Supported Platforms

Platform	Method	Free Quota	Status
X/Twitter	syndication API (no authentication required)	Unlimited	✅
YouTube	RSS Feed (channel level) + yt-dlp	Unlimited	✅
Bilibili	Public API (ranking/v2)	Unlimited	✅
GitHub	Search API (no authentication)	60 times/hour	✅
Reddit	PullPush API	Unlimited	✅
Douyin	Trending Search API (no authentication required)	Unlimited	✅
LinuxDo	Discourse JSON API	Unlimited	⏳ Requires login session
Xiaohongshu	Web-side SSR parsing	Unlimited	⏳ Requires login session
WeChat Official Accounts	Sogou WeChat Search	Limited	⏳ Requires login session
WeChat Channels	No public API	-	⏳ Requires Playwright

Usage

# 采集所有平台
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py

# 采集单个平台
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source twitter
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source youtube
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source bilibili
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source github
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source reddit
python3 ~/clawd/skills/content-source-aggregator/scripts/fetch_all.py --source linuxdo

Output

Write standardized JSON to ~/clawd/workspace/content-pipeline/hotpool/YYYY-MM-DD.json

{
  "date": "2026-02-19",
  "fetched_at": "2026-02-19T10:00:00+08:00",
  "items": [
    {
      "source": "reddit",
      "title": "...",
      "url": "...",
      "summary": "...",
      "heat_score": 85,
      "category": "AI/Tech",
      "engagement": {"upvotes": 1200, "comments": 340},
      "fetched_at": "..."
    }
  ]
}

Platform API Details

X/Twitter

• Endpoint: https://syndication.twitter.com/srv/timeline-profile/screen-name/{username}
• Method: Parse the tweet text in the returned HTML
• Configurable list of accounts to follow

YouTube

• Endpoint: https://www.youtube.com/feeds/videos.xml?channel_id={id}
• Method: RSS XML parsing
• Configurable list of channels to follow
• Alternative: yt-dlp to get trending

Bilibili

• Trending ranking: https://api.bilibili.com/x/web-interface/ranking/v2?rid=0&type=all
• Trending search: https://api.bilibili.com/x/web-interface/wbi/search/square?limit=10
• Requires Referer: https://www.bilibili.com

GitHub

• Endpoint: https://api.github.com/search/repositories?q=stars:>100+pushed:>{date}&sort=stars
• No authentication, 60 times/hour, sufficient

Reddit

• Endpoint: https://api.pullpush.io/reddit/search/submission/?subreddit={sub}&sort=score&sort_type=desc&size=10
• Free and unlimited

LinuxDo

• Endpoint: https://linux.do/latest.json?order=default
• Discourse standard API, free and public

Configuration

Edit scripts/config.json to customize the accounts/channels/subreddits to follow.

Integration with the Content Pipeline

This skill is Phase 1 (hotspot collection) of the content creation SOP, and its output is used by the research agent for topic scoring.