SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsxiaohongshu-fetch

xiaohongshu-fetch

Office & productivity

View repository

Updated 10-16 00:22

About this skill

Xiaohongshu scraping and saving

One-click scraping script (preferred)

Use the local script to complete scraping first; no need to manually curl and parse the page:

python3 xiaohongshu-fetch/scripts/fetch.py "<小红书笔记链接>" [输出目录]

Default output directory: ~/Downloads/xhs_<note_id>

Output files:

  • xhs_<note_id>.html: webpage snapshot
  • xhs_<note_id>.metadata.json: title, body text, author, publish time, engagement data, tags, chapters, media URLs, and other metadata
  • Video notes:xhs_<note_id>.mp4、xhs_<note_id>.<字幕语言>.srt、xhs_<note_id>.transcript.txt
  • Image-text notes:images/ Save images under

Quickly check whether the link can be parsed:

python3 xiaohongshu-fetch/scripts/fetch.py "<小红书笔记链接>" [输出目录] --skip-media

The script will prioritize parsing the window.__INITIAL_STATE__in the webpage, automatically handling undefined、video.mediaV2 / video.consumer.mediaV2, video masterUrl/backupUrls, subtitles, and image URLs. When run repeatedly, existing large videos/subtitles/images will be automatically skipped. Only when the script fails, proceed to the manual scraping/saving process below.

Two pathways

After the user sends a Xiaohongshu link, take different paths based on the note type and user intent:

用户发链接
    │
    ├── 用户说"沉淀"或"入库"
    │       │
    │       ├── 视频笔记 → 视频对标库（AppToken/TableID 从私有配置读取）
    │       │       ① 下载视频 → ② volc-asr 转写 → ③ LLM分段 → ④ 写入bitable → ⑤ 按用户许可清理临时文件
    │       │
    │       └── 图文笔记 → 社媒爆款选题库（AppToken/TableID 从私有配置读取）
    │               若用户指定其他目标 → 按指定去沉淀
    │
    └── 用户只发链接不提沉淀
            抓取元数据+图片，输出结构化摘要，不写入bitable

Configuration reading

Do not hardcode Cookie, AppToken, TableID, or Feishu links into AGENTS, skill, regular memory, or logs.

Before executing writes, inject the target table configuration from the shared private configuration runner. Do not directly read or print the entire secrets file:

$HOME/.config/agent-secrets/agent_secrets.py check --profile xiaohongshu
$HOME/.config/agent-secrets/agent_secrets.py run --profile xiaohongshu -- <command>

Claude Code and Codex use the same $HOME/.config/agent-secrets/secrets.json; the runner only injects the specified profile into the subprocess. Corresponding environment variable names:

Purpose AppToken environment variable TableID environment variable
Video Benchmark Library XHS_VIDEO_BENCHMARK_APP_TOKEN XHS_VIDEO_BENCHMARK_TABLE_ID
Social Media Viral Topic Library XHS_SOCIAL_TOPIC_APP_TOKEN XHS_SOCIAL_TOPIC_TABLE_ID
Learning Library XHS_LEARNING_APP_TOKEN XHS_LEARNING_TABLE_ID

If the current environment is not configured, first explain which item is missing; do not guess from historical messages or regular memory.

Pathway 1: Video notes → Video Benchmark Library

Target bitable: Video Benchmark Library. AppToken and TableID are read from private configuration or environment variables, and are not hardcoded in the skill.

Step 1: Scrape the page, extract the direct video URL and metadata

Prefer using the "one-click scraping script". Refer to this section only for manual handling.

Refer to "Core Methods" to extract:

  • title、desc、user.nickname、time、interactInfo
  • video.media.stream.h264[0].masterUrl(direct video URL)

Step 2: Download the video

Once video.media.stream.h264[0].masterUrlhas been obtained from Step 1, download directly with curl:

curl -L -o "/tmp/video.mp4" "<视频直链>"

Download path:/tmp/video.mp4

Step 3: Call volc-asr for transcription (key step)

You must call the locally or cloud-installed volc-asr skill; the transcription script is located at:

  • In the repository:python3 volc-asr/scripts/transcribe.py '<视频文件路径>'
  • Installed skill:python3 <skills-dir>/volc-asr/scripts/transcribe.py '<视频文件路径>'

Before execution, ensure the environment variables are configured:

export TOS_ACCESS_KEY="..."
export TOS_SECRET_KEY="..."
export TOS_BUCKET="..."
export VOLC_ASR_TRIAL_APP_ID="..."
export VOLC_ASR_TRIAL_TOKEN="..."

Do not write real AppToken, TableID, Cookie, TOS keys, or ASR Token into the skill, repository, or regular logs.

Step 4: LLM reads the transcription result and splits it into paragraphs

After transcription is complete, the LLM reads the .txt file and outputs JSON in the following format:

{
  "segments": [
    {
      "content": "段落原文",
      "summary": "一句话摘要",
      "position": "开头|中间|高潮|结尾",
      "viral_hook": true,
      "hook_type": "悬念|冲突|数字|情感|null"
    }
  ],
  "structure_analysis": "整体结构一句话描述",
  "opening_analysis": "开头分析",
  "viral_points": ["爆点1", "爆点2"]
}

Step 5: Write to the Video Benchmark Library bitable

Field mapping:

bitable field Value source
Source Link Original note URL
Platform Xiaohongshu
Author user.nickname
Publish Date time (timestamp → date format)
Play Count interactInfo view count (fill in if available)
Like Count interactInfo likedCount
Favorite Count interactInfo.collectedCount
Comment Count interactInfo.commentCount
Share Count interactInfo.shareCount
Body Voiceover Script Concatenate all paragraph content in order
Voiceover Script Analysis structure_analysis + opening_analysis + viral_points summary
Content Type Determine based on body: knowledge/tips, voiceover product recommendation, story narrative, opinion output, review comparison, tutorial sharing, trending topic commentary, other

Write API:

POST https://open.feishu.cn/open-apis/bitable/v1/apps/{VIDEO_BENCHMARK_APP_TOKEN}/tables/{VIDEO_BENCHMARK_TABLE_ID}/records

Step 6: Clean up temporary files

After completing transcription and writing to bitable, explain the locations of the temporary video and transcription text, and ask the user whether to clean up. Only clean up after obtaining explicit permission, and prioritize moving to trash; do not directly rm -f。

Path Two: Image-text notes → Social media viral topic library (default)

Target bitable: Social media viral topic library. AppToken and TableID are read from private configuration or environment variables, not hardcoded in the skill.

Step 1: Scrape the page, extract metadata

Prioritize using the "one-click scraping script". Refer to this section only when handling manually.

Refer to "Core Methods" to extract: title, desc, tagList, user.nickname, time, interactInfo

Step 2: Perform OCR/description on each image

For each image in imageList:

  • Send to LLM for in-image text recognition + content summary
  • Merged output: page-by-page summary + full-text restoration version

Step 3: Write to the social media viral topic library bitable

Field mapping:

bitable field Value source
Social media viral topic library Title (primary field)
Title title
Source Link Original note URL
Platform Xiaohongshu
Author user.nickname
Date time
One-sentence summary LLM generates based on desc + image content
Article Structure LLM analyzes the image-text structure
Inspiration Points LLM extracts core inspiration
Content Type Determine based on body
Tags Take the first 5 from tagList

Write API:

POST https://open.feishu.cn/open-apis/bitable/v1/apps/{SOCIAL_TOPIC_APP_TOKEN}/tables/{SOCIAL_TOPIC_TABLE_ID}/records

Path Two (alternative): User-specified storage target

If the user says "save to XXX library" or "add to XXX", create a record according to the user-specified bitable.

Platform Identification

Domain Platform
xiaohongshu.com Xiaohongshu
douyin.com Douyin
bilibili.com Bilibili
weibo.com Weibo
mp.weixin.qq.com WeChat Official Account
zhihu.com Zhihu
youtube.com YouTube

Core Scraping Methods

URL and Authentication Context

  • URL format:https://www.xiaohongshu.com/explore/{note_id}?xsec_token=...&xsec_source=pc_user
  • When accessing HTML, a valid login-state Cookie is usually required, along with xsec_token
  • Login-state Cookies are sensitive credentials; do not write them into AGENTS, ordinary memory, public documents, or logs

Method A: Read the webpage meta

curl -L -A 'Mozilla/5.0' '<笔记URL>'

Extract: og:title, description, og:image, og:video

Method B: Parse window.__INITIAL_STATE__(primary)

Key paths:root.note.noteDetailMap.<note_id>.note or note.noteDetailMap.<note_id>.note Prioritize checking: title, desc, type, tagList, user.nickname, interactInfo, video.media.stream, imageList For direct video links, prioritize checking:video.media.stream.h264[0].masterUrl

Decision rules

  • type == "video" → video note, use pipeline one
  • type == "normal" or imageList has content → image-text note, use pipeline two
  • interactInfo all empty → may be a non-public note; mark "interaction data not public"

Failure handling

  • Xiaohongshu risk-control blocking → change User-Agent or add delay and retry
  • The page opens but the script reports no INITIAL_STATE → re-copy the complete link with xsec_token ; if necessary, log in to the browser and then copy the link
  • Direct video link 404 → try backupUrls
  • Direct video link expired → rerun the one-click scraping script to refresh metadata.json and the direct link
  • No subtitles → after downloading the video, call volc-asr for transcription
  • volc-asr call fails → record the error and prompt the user that they can transcribe manually and then use the "body narration script" field
  • bitable write fails → print the complete error response and prompt to check the token and table permissions

One-sentence workflow

Video notes: scrape page → curl download direct video link → volc-asr transcription → LLM segmentation → write to video benchmark library → clean up temporary files with user permission Image-text notes: scrape page → image OCR → LLM summary → write to social media topic library (default) or a specified table