content-collector

Content Collector

Auto-collect social media content → AI summarize → Save to Feishu bitable.

Quick Reference

Trigger	Action
X/Twitter link	x-tweet-fetcher skill
WeChat article	web-content-fetcher (Scrapling)
Other platforms	defuddle → fallback to baoyu-url-to-markdown
Screenshot	OCR → extract URL → collect

Workflow

Link/Screenshot → Platform Detect → Dedupe → Extract → Summarize → Save to Bitable

Step 1: Platform Detection

python3 scripts/extract_content.py "<url>"

Returns: platform_id, skill to use, fallback_skills, CSS selectors.

See references/platforms.md for full platform mapping.

Step 2: Deduplication

python3 scripts/deduplicate.py "<url>"           # Check if exists
python3 scripts/deduplicate.py --add "<url>"     # Add to cache after saving

Step 3: Extract Content

Call the skill returned by Step 1:

• X/Twitter: Use x-tweet-fetcher skill
• WeChat: Use web-content-fetcher skill (Scrapling)
• Others: Use defuddle skill

Step 4: AI Summarize + Independent Tag Generation (v2.0)

Updated 2026-03-21: Uses independent generation mode, no longer relies on historical tag pool

Core Principles：

• Independent Generation: Generate tags only based on the current article content, without referencing historical tags
• Fixed Structure: Object (2) + Scenario (1) + Type (1) + Method (1) = 5 tags
• Format Specification: English lowercase + hyphen connection, concise Chinese phrases

---

4.1 Extract Article Content

Article content obtained from Step 3 (title, body, source, etc.)

---

4.2 Call Model to Generate 5 Tags

Prompt Template：

请阅读以下内容，并严格按照"对象、场景、类型、方法"四类标签体系输出 5 个标签。

【标签体系】
- 对象（2个）：内容涉及的核心主体/技术/工具，如 openclaude、agent、mcp、prompt、浏览器自动化、记忆系统、量化交易、学习资源、实战案例、产品思考
- 场景（1个）：内容应用的实际场景/用途，如 投资分析、自动化测试、知识管理、代码生成
- 类型（1个）：内容的表现形式，如 技术教程、实战案例、产品分析、工具推荐、观点分享
- 方法（1个）：内容涉及的方法论/技巧，如 工作流、评测、prompt优化、架构设计

【规则】
1. 对象 2 个，场景 1 个，类型 1 个，方法 1 个，共 5 个
2. 不参考历史标签，只根据当前文章内容生成
3. 英文标签小写，多个单词用 `-` 连接（如 claude-code）
4. 中文标签使用简洁固定短语（如 投资分析、技术教程）
5. 不输出空泛标签，如 AI、工具、技术、效率
6. 只输出 JSON，不输出解释

【输出格式】
{
  "tags": {
    "对象": ["标签1", "标签2"],
    "场景": ["标签3"],
    "类型": ["标签4"],
    "方法": ["标签5"]
  }
}

【内容】
{文章内容}

---

4.3 Tag Normalization

Normalize the tags output by the model:

def normalize_tag(tag):
    # 1. 去掉首尾空格
    tag = tag.strip()
    
    # 2. 英文转小写
    tag = tag.lower()
    
    # 3. 英文多个单词用 `-` 连接
    # 如 "Claude Code" -> "claude-code"
    if ' ' in tag and tag.replace(' ', '').isalpha():
        tag = '-'.join(tag.split())
    
    # 4. 去重（同一类别内）
    return tag

---

4.4 Tag Validation

Validate the normalized tag structure:

def validate_tags(tags):
    """
    校验规则：
    - 对象恰好 2 个
    - 场景恰好 1 个
    - 类型恰好 1 个
    - 方法恰好 1 个
    - 总数恰好 5 个
    - 标签无重复（跨类别也检查）
    """
    errors = []
    
    # 检查各类别数量
    if len(tags.get("对象", [])) != 2:
        errors.append(f"对象需要 2 个，当前 {len(tags.get('对象', []))} 个")
    if len(tags.get("场景", [])) != 1:
        errors.append(f"场景需要 1 个，当前 {len(tags.get('场景', []))} 个")
    if len(tags.get("类型", [])) != 1:
        errors.append(f"类型需要 1 个，当前 {len(tags.get('类型', []))} 个")
    if len(tags.get("方法", [])) != 1:
        errors.append(f"方法需要 1 个，当前 {len(tags.get('方法', []))} 个")
    
    # 检查总数
    total = sum(len(v) for v in tags.values())
    if total != 5:
        errors.append(f"标签总数需要 5 个，当前 {total} 个")
    
    # 检查重复（跨类别）
    all_tags = []
    for v in tags.values():
        all_tags.extend(v)
    if len(all_tags) != len(set(all_tags)):
        errors.append("存在重复标签")
    
    return errors

Retry Mechanism: If validation fails, allow the model to retry once

---

4.5 Write to Feishu

Write the final 5 tags to the Feishu Bitable (flattened into a string array):

feishu_bitable_app_table_record(
  action="create",
  app_token="ND8ObCuSya5Dv3sREZYc03Ilngh",
  table_id="tblaHDM5kjtikIl9",
  fields={
    "标签": ["openclaude", "agent", "投资分析", "实战案例", "工作流"]
  }
)

---

Complete Flow (v2.0)

1. 提取文章内容（Step 3）
      ↓
2. 调用模型生成 5 个标签（固定结构）
      ↓
3. 规范化标签（小写、去空格、连字符）
      ↓
4. 校验标签结构（对象2 + 场景1 + 类型1 + 方法1）
      ↓
5. 写入飞书

---

4.6 Tag System Reference (for model reference only, not involved in matching)

Category	Tag Examples
Object	openclaude, agent, mcp, prompt, browser automation, memory system, quantitative trading, claude-code, workflow, evaluation
Scenario	investment analysis, automated testing, knowledge management, code generation, data processing, dialogue system, content creation
Type	technical tutorial, practical case, product analysis, tool recommendation, opinion sharing, industry insight
Method	workflow, evaluation, prompt optimization, architecture design, performance optimization, security hardening

Note: This solution no longer uses a historical tag pool, does not perform fuzzy matching, semantic approximate matching, or old tag attachment

• Quantity ≤ 5
• Used tags from the existing pool or automatically created

Step 5: Save to Feishu Bitable

Required fields:

• 标题 - Article title
• 来源 - Source platform (X/Twitter, WeChat Official Account, etc.)
• 分类 - Content category (🔧Tool Recommendation/📖Technical Tutorial/🛠️Practical Case/💡Product Idea)
• 摘要内容 - AI-generated content summary
• 原文链接 - Original URL (URL type)
• 原文文件 - Feishu cloud space file link (URL type)
• 标签 - 5 tag array

Complete flow (v2.2 - forced script solution):

⚠️ Important: Must pass through save_to_bitable.py script to write, direct calls are prohibited feishu_bitable_app_table_record

1. Extract content using platform-specific skill (x-tweet-fetcher, web-content-fetcher, etc.)

2. Generate summary - AI summarize the content

3. Generate tags - 5 tags (Object 2 + Scenario 1 + Type 1 + Method 1)

4. Save as local .md file - Full content preserved to /tmp/content.md

5. Force use of script to write - Direct tool calls prohibited:

   python3 scripts/save_to_bitable.py \
       --title "文章标题" \
       --source "X/Twitter" \
       --category "🛠️实战案例" \
       --url "https://x.com/i/status/..." \
       --content-file /tmp/content.md
   

   The script will automatically complete:

   • Upload file to Feishu cloud space
   • Obtain real file_token
   • Write to the "Original File" field (only written when upload succeeds)
   • Write to the "Original Link" field
   • Return record ID

6. Update dedupe cache

   python3 scripts/deduplicate.py --add "<url>"
   

🚫 Prohibited Actions:

• Direct calls prohibited feishu_bitable_app_table_record Write to "Original File"
• Prohibited use of placeholder URLs (http://查看完整内容, http://推文链接 etc.)
• Do not assume a file link when no file has been uploaded

✅ Mandatory check:

• The script verifies the file upload status
• Only when upload_success=True is "Original File" written
• An upload failure will raise an error and will not write false data

Important: Always save BOTH 原文链接 (original URL) and 原文文件 (Feishu Drive backup). This ensures content remains accessible even if the original link becomes unavailable.

Changelog v2.2 (2026-03-31):

• ✅ Mandatory script approach: Must be written via save_to_bitable.py , direct calls to feishu_bitable_app_table_record
• ✅ URL format validation: Automatically intercepts placeholders such as http://查看完整内容 and other placeholders
• ✅ Upload verification: "Original File" is written only when the upload actually succeeds

Changelog v2.1 (2026-03-29):

• ✅ Fixed: On upload failure, no false URL is written, ensuring the 原文文件 field contains only files that actually exist
• ✅ Added: Upload status verification; on failure, an error is logged but no placeholder is written
• ✅ Improved: Stricter field write conditions to prevent test data/false links from entering the table

Changelog v2.0 (2026-03-29):

• ✅ Fixed: save_to_bitable.py now uploads files to Feishu Drive before creating records
• ✅ Added: upload_file_to_feishu() function handles file upload with proper multipart/form-data
• ✅ Changed: Records now include 原文文件 field with cloud storage URL instead of inline content
• ⚠️ Note: Previous versions missed the upload step, causing empty 原文文件 fields

Bitable config: See references/feishu_config.md or use environment variables:

• FEISHU_BITABLE_APP_TOKEN
• FEISHU_BITABLE_TABLE_ID

Scripts

Script	Purpose
scripts/extract_content.py	Platform detection + skill routing
scripts/deduplicate.py	URL deduplication (cache + document check)
scripts/append_to_feishu.py	Format content for Feishu doc (backup)
scripts/ocr_image.py	OCR for screenshots (optional)

Dependencies

Required Skills

• feishu-doc / feishu-bitable - Read/write Feishu
• defuddle - Generic web extraction

Platform-Specific (install as needed)

• x-tweet-fetcher - X/Twitter
• web-content-fetcher - WeChat (Scrapling)
• baoyu-url-to-markdown - Fallback

Optional

• pytesseract + tesseract-ocr - Local OCR

Configuration

Set via environment variables or see references/feishu_config.md:

export FEISHU_BITABLE_APP_TOKEN="your_app_token"
export FEISHU_BITABLE_TABLE_ID="your_table_id"

References

• references/platforms.md - Full platform mapping and selectors
• references/feishu_config.md - Feishu bitable configuration
• references/tagging_spec.md - ⚠️ Disabled (see Step 4 v2.0)