SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscontent-collector

content-collector

General & other

View repository

Updated 10-16 00:00

About this skill

Content Collector - Social Content Collection Assistant

Automatically collects great social media content, organizes it with AI, and stores it in Feishu Docs.

Trigger Conditions

Automatically triggered when the user sends the following:

  • Link: X/Twitter, Jike, WeChat Official Accounts, Reddit, Zhihu, Bilibili, Hacker News, etc.
  • Image: Screenshot (extract links after OCR recognition)
  • Mixed: Link + screenshot

Dependent Skills

Important: This skill does not directly call external scripts or paths; instead, it outputs guidance, and the Agent calls the corresponding skill at runtime. The following are the skills actually used in the current environment; please ensure they are installed.

Required Skills

Skill Purpose Installation Location
feishu-doc Read and write Feishu Docs (append content, read existing content for deduplication) ~/.agents/skills/feishu-doc*
defuddle General web page body extraction (WeChat Official Accounts, Jike, Reddit, Zhihu, etc.) ~/.opencode/skills/defuddle/

Recommended Skills (by platform)

Skill Purpose Installation Location
x-tweet-fetcher X/Twitter tweets, articles, timeline extraction (zero-dependency FxTwitter API) ~/.opencode/skills/x-tweet-fetcher/
web-content-fetcher WeChat Official Account scraping (Scrapling solution, perfectly bypasses anti-scraping) ~/.openclaw/workspace/skills/web-content-fetcher/
baoyu-url-to-markdown General URL to Markdown (fallback solution) ~/.opencode/skills/baoyu-url-to-markdown/

Optional Skills

Skill Purpose
feishu-cron-reminder Daily scheduled reminder to check collected content
feishu-send-file Send collected export files to a Feishu group

Optional Local Dependencies

Dependency Purpose Installation Method
pytesseract + pillow Local image OCR pip install pytesseract pillow
tesseract-ocr OCR engine macOS: brew install tesseract; Linux: apt-get install tesseract-ocr tesseract-ocr-chi-sim

Workflow

用户发送链接/图片
  → 平台检测 (extract_content.py)
  → 去重检查 (deduplicate.py)
  → Agent 调用对应 skill 提取内容
  → AI 整理成结构化格式
  → 格式化 (append_to_feishu.py)
  → Agent 调用 feishu-doc 追加到文档

Step 1: Platform Detection

Run extract_content.py to detect the link source and obtain the recommended skill and extraction method:

python3 scripts/extract_content.py "https://x.com/user/status/123"

Example output:

{
  "platform_id": "twitter",
  "platform_label": "X/Twitter",
  "url": "https://x.com/user/status/123",
  "skill": "x-tweet-fetcher",
  "fallback_skills": [],
  "note": "使用 x-tweet-fetcher skill 提取推文/文章内容"
}

Supported Platform Mapping：

Platform Domain Preferred Skill Fallback
X/Twitter x.com, twitter.com x-tweet-fetcher —
WeChat Official Accounts mp.weixin.qq.com web-content-fetcher (Scrapling) defuddle
Jike okjike.com, jike.cn defuddle baoyu-url-to-markdown
Reddit reddit.com defuddle baoyu-url-to-markdown
Hacker News news.ycombinator.com defuddle —
Zhihu zhihu.com defuddle baoyu-url-to-markdown
Bilibili bilibili.com, b23.tv defuddle baoyu-url-to-markdown
Other * defuddle baoyu-url-to-markdown

Step 2: Deduplication Check

python3 scripts/deduplicate.py "https://x.com/user/status/123"

Check methods:

  1. Local cache：.cache/collected_urls.json
  2. Document check: Pass in the Feishu document content file for matching
  3. URL normalization: Automatically remove tracking parameters such as utm_source, fbclid, etc.

Step 3: Content Extraction

Based on the output of Step 1, the Agent calls the corresponding skill to extract content. For example:

  • X/Twitter: Call x-tweet-fetcher skill
  • WeChat Official Accounts: Call defuddle skill
  • Other platforms: Call defuddle or baoyu-url-to-markdown skill

The Agent is responsible for calling skills; this skill's scripts only perform detection and formatting, and do not directly call external skills.

Platform-specific tips(embedded in the selectors field of extract_content.py):

  • WeChat Official Accounts CSS selectors：#activity-name(title),#js_name(author),#js_content(body),#publish_time(time)
  • Immediately CSS selectors：meta[property="og:title"](author),meta[property="og:description"](content),time(time)
  • Reddit: Supports JSON API, append .json to the end of the URL to get structured data

Step 4: AI Organization

After extracting the raw content, the AI organizes it into structured information:

  • Title: content title
  • Source: author/platform
  • Category: categorized by content type
  • Summary content: AI-generated summary, summarizing the core content in 3-5 sentences
  • Original link: original link
  • Original file: Feishu cloud space .md file link

Step 5: Save to Bitable (recommended)

Optimized process（v1.4.0）：

1. 抓取内容 → 保存为 .md 文件
2. 上传到飞书云空间 → 获取文件链接
3. LLM 只输出短字段（标题、来源、分类、摘要内容）
4. 写入多维表格，原文通过文件链接跳转

Agent workflow：

  1. Call the corresponding skill to scrape content
  2. Save as a local .md file (e.g. /tmp/content_{id}.md）
  3. Call feishu_drive_file upload to upload to cloud space
  4. Get the file link (e.g. https://my.feishu.cn/file/xxx）
  5. Call feishu_bitable_app_table_record to write to Bitable

Advantages：

  • ✅ Cost-saving: the LLM does not output long text content
  • ✅ Stable: file content is complete and will not be truncated
  • ✅ Viewable: open .md files directly in Feishu
  • ✅ Searchable: summary content + title are searchable

Field structure：

Field Type Purpose
Title Text Searchable
Source Text Filterable
Category Single select Filterable
Original link Hyperlink Original source
Summary content Text AI-generated summary, searchable
Record time Created time Automatically recorded
Original file Hyperlink Feishu cloud space .md file

Bitable configuration：

  • App Token: ND8ObCuSya5Dv3sREZYc03Ilngh
  • Table ID: tblaHDM5kjtikIl9

Step 5 (alternative): Format as a Markdown document

If you need to save to a regular Feishu document (not Bitable), use:

python3 scripts/append_to_feishu.py '<json_content>'

Output as formatted Markdown, for the Agent to call feishu-doc's append operation to write to the document.

Step 6: Image OCR (optional)

When the user sends a screenshot instead of a link:

python3 scripts/ocr_image.py /path/to/image.png
  • If pytesseract is installed locally → perform OCR directly
  • Otherwise → return the image path and prompt the Agent to use an external OCR service
  • After recognition, automatically extract the URL from the text and return to Step 1

Script list

Script Purpose Directly usable
scripts/extract_content.py Platform detection + skill routing ✅ Runs independently
scripts/deduplicate.py URL deduplication (local cache + document matching) ✅ Runs independently
scripts/append_to_feishu.py Format content as Feishu Markdown (alternative) ✅ Runs independently
scripts/ocr_image.py Image OCR (requires pytesseract or an external service) ⚠️ Optional dependency required

Note: No longer needed after v1.4.0 save_to_bitable.py, switched to the Feishu Cloud Space upload approach.

Deduplication mechanism

URL normalization

Automatically removes the following tracking parameters:

  • utm_source, utm_medium, utm_campaign, utm_term, utm_content
  • fbclid, gclid
  • ref, source

Cache storage

.cache/collected_urls.json
{
  "https://x.com/user/status/123": {
    "original_url": "https://x.com/...",
    "date": "2026-03-14T10:00:00",
    "metadata": {...}
  }
}

Configuration

Feishu document Token: A document is created on first use, and subsequent entries are appended to the same document. Suggested document title:社交内容收藏 - 每日精选

Daily reminder(optional, requires the feishu-cron-reminder skill): Every day at 18:00, push the day's bookmark count, title list, and document link.

Changelog

v1.4.0 (2026-03-18)

  • 🎉 Long text optimization: The original text is uploaded to Feishu Cloud Space, and the link is stored in the URL field
  • ✅ Added the "Original File" field, renamed "Original Content" to "Summary Content"
  • ✅ The LLM no longer outputs long text, greatly saving tokens
  • ✅ The original text can be viewed by jumping through the Feishu file link, supporting the .md format

v1.3.0 (2026-03-17)

  • 🎉 WeChat Official Account scraping capability upgrade: Switched to the Scrapling approach, perfectly bypassing WeChat anti-scraping
  • 🔧 Updated platform mapping: WeChat Official Account prefers the web-content-fetcher skill
  • 🔧 Added the scrapling_command field to directly output the scraping command

v1.2.0 (2026-03-14)

  • 🔧 Removed all hardcoded paths to adapt to multi-environment deployment
  • 🔧 Rewrote extract_content.py as a platform detection + skill routing pattern
  • 🔧 Deleted empty-shell extraction scripts (extract_jike/weixin/reddit.py), with selector information embedded into extract_content.py
  • 🔧 Updated dependency declarations to the skills actually used in the current environment
  • 🔧 Fixed type annotation and bare except issues

v1.1.0 (2026-03-14)

  • ✅ Added deduplication mechanism (local cache + document check)
  • ✅ Support for more platforms (dedicated extractors for Jike, WeChat Official Account, and Reddit)
  • ✅ Added image OCR recognition script
  • ✅ Optimized Feishu document format (automatic numbering, interaction data formatting)

v1.0.0 (2026-03-14)

  • Initial version
  • Supports X/Twitter, Jike, and WeChat Official Account