SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsonline-content-collector

online-content-collector

Office & productivity

View repository

Updated 10-15 18:46

About this skill

Skill: Online Content Collector

Overview

This skill is designed to fully automate the process from "discovering links" to "local archiving". It scans links with specific tags in an Obsidian vault, aggregates them, and downloads them as complete local Markdown archives containing text, images, videos, and attachments.

Core Configuration Parsing

This skill relies on the user's Vault root directory's AGENTS.md。The Agent must first read this file before executing the script, and parse the following paths:

  • 下载列表目录位于: used to pass to --list-dir。
  • 下载内容目录位于: used to pass to --archive-dir。

Core Workflow

Phase 1: Discovery & Aggregation

  1. Configuration Reading: The Agent reads AGENTS.md, to determine the target path.
  2. Execute Script: Call scripts/collect_links.py, pass in --vault-path and --list-dir。
  3. Metadata Extraction and Update: The script scans files containing #Marker-待下载 , extracts time and source, generates a YAML-format task list [yyyy-mm-dd-hh 下载列表整理.md], and updates the original file tag to #Marker-下载中-YYYYMMDD。

Phase 2: User Confirmation

  1. Stop and Check: The Agent outputs the YAML list file path and waits for user confirmation.

Phase 3: Execution & Archival

  1. Execute Script: Call scripts/process_downloads.py, pass in --list-file and --archive-dir。
  2. Task Processing and Dispatch:
    • YouTube / X (Twitter): Use yt-dlp to download. Request the highest quality, must download and retain the full JSON metadata (--write-info-json）。
    • Unknown Site: If the domain cannot be recognized or no download method is configured, mark it directly as "Download Failed (Unrecognized Site)" without attempting.
  3. Isolation Directory Creation: Create an independent directory for each download task, naming convention:[YYYY-MM-DD-HH] {分类} {描述/ID}。
  4. Content Localization:
    • Main Document: Create a file with the same name in the directory .md .
    • Asset Storage: All .mp4, .json, .jpg and other assetsare all stored in the task directory。
    • Reference Linking: In the Markdown file, use local relative paths to link to videos in the same directory.

Phase 4: Reporting & Closing

  1. List Write-back: The script updates the status in the YAML list to "Download Complete" or "Download Failed".
  2. Tag Synchronization: Based on the script output, the Agent updates the link tag in the original file to #Marker-已下载-YYYYMMDD。

Dependencies

  • yt-dlp: Video capture.
  • MarkItDown / Pandoc: Web page to Markdown.
  • ffmpeg: Video merging.

Best Practices

  • Path Alignment: Always read paths from agents.md , do not hardcode.
  • Metadata Retention: Record the original URL and collection time in the header of the clipped Markdown.
  • Exception Fault Tolerance: When a download fails, record the reason for the error without interrupting subsequent tasks.