yichen-xiaohongshu-fetch

Xiaohongshu Scraping and Saving

One-Click Scraping Script (Preferred)

Use the local script to complete scraping first; no need to manually curl and parse the page:

python3 yichen-xiaohongshu-fetch/scripts/fetch.py "<小红书笔记链接>" [输出目录]

Default output directory: ~/Downloads/xhs_<note_id>

Output files:

• xhs_<note_id>.html: Webpage snapshot
• xhs_<note_id>.metadata.json: Metadata such as title, body text, author, publish time, engagement data, tags, chapters, media URLs, etc.
• Video notes:xhs_<note_id>.mp4、xhs_<note_id>.<字幕语言>.srt、xhs_<note_id>.transcript.txt
• Image-text notes:images/ Save images under

Quickly check whether a link can be parsed:

python3 yichen-xiaohongshu-fetch/scripts/fetch.py "<小红书笔记链接>" [输出目录] --skip-media

The script will prioritize parsing the window.__INITIAL_STATE__in the webpage, automatically handling undefined、video.mediaV2 / video.consumer.mediaV2, video masterUrl/backupUrls, subtitles, and image URLs. When run repeatedly, existing large videos/subtitles/images are automatically skipped. Only when the script fails, proceed to the manual scraping/saving process below.

Two Pipelines

After the user sends a Xiaohongshu link, follow different paths based on the note type and user intent:

用户发链接
    │
    ├── 用户说"沉淀"或"入库"
    │       │
    │       ├── 视频笔记 → 视频对标库（AppToken/TableID 从私有配置读取）
    │       │       ① 下载视频 → ② yichen-volc-asr 转写 → ③ LLM分段 → ④ 写入bitable → ⑤ 按用户许可清理临时文件
    │       │
    │       └── 图文笔记 → 社媒爆款选题库（AppToken/TableID 从私有配置读取）
    │               若用户指定其他目标 → 按指定去沉淀
    │
    └── 用户只发链接不提沉淀
            抓取元数据+图片，输出结构化摘要，不写入bitable

---

Configuration Reading

Do not hardcode Cookie, AppToken, TableID, or Feishu links into AGENTS, skill, ordinary memory, or logs.

Before executing a write, inject the target table configuration from the shared private configuration runner. Do not directly read or print the entire secrets file:

$HOME/.config/agent-secrets/agent_secrets.py check --profile xiaohongshu
$HOME/.config/agent-secrets/agent_secrets.py run --profile xiaohongshu -- <command>

Claude Code and Codex use the same $HOME/.config/agent-secrets/secrets.json; the runner only injects the specified profile into the subprocess. Corresponding environment variable names:

Purpose	AppToken environment variable	TableID environment variable
Video Benchmark Library	XHS_VIDEO_BENCHMARK_APP_TOKEN	XHS_VIDEO_BENCHMARK_TABLE_ID
Social Media Viral Topic Library	XHS_SOCIAL_TOPIC_APP_TOKEN	XHS_SOCIAL_TOPIC_TABLE_ID
Learning Library	XHS_LEARNING_APP_TOKEN	XHS_LEARNING_TABLE_ID

If the current environment is not configured, first state which item is missing; do not guess from historical messages or ordinary memory.

---

Pipeline One: Video Notes → Video Benchmark Library

Target bitable: Video Benchmark Library. AppToken and TableID are read from private configuration or environment variables, and are not hardcoded in the skill.

Step 1: Scrape the page, extract the video direct link and metadata

Prefer using the "One-Click Scraping Script". Refer to this section only for manual handling.

Refer to "Core Methods" to extract:

• title、desc、user.nickname、time、interactInfo
• video.media.stream.h264[0].masterUrl(video direct link)

Step 2: Download the video

Once video.media.stream.h264[0].masterUrlis obtained from Step 1, directly curl to download:

curl -L -o "/tmp/video.mp4" "<视频直链>"

Download path:/tmp/video.mp4

Step 3: Call yichen-volc-asr for transcription (key step)

You must call the locally or cloud-installed yichen-volc-asr skill; the transcription script is located at:

• In the repository:python3 yichen-volc-asr/scripts/transcribe.py '<视频文件路径>'
• Installed skill:python3 <skills-dir>/yichen-volc-asr/scripts/transcribe.py '<视频文件路径>'

Ensure environment variables are configured before execution:

export TOS_ACCESS_KEY="..."
export TOS_SECRET_KEY="..."
export TOS_BUCKET="..."
export VOLC_ASR_TRIAL_APP_ID="..."
export VOLC_ASR_TRIAL_TOKEN="..."

Do not write real AppToken, TableID, Cookie, TOS keys, or ASR Token into the skill, repository, or ordinary logs.

Step 4: LLM reads the transcription result and splits it into paragraphs

After transcription is complete, the LLM reads the .txt file and outputs JSON in the following format:

{
  "segments": [
    {
      "content": "段落原文",
      "summary": "一句话摘要",
      "position": "开头|中间|高潮|结尾",
      "viral_hook": true,
      "hook_type": "悬念|冲突|数字|情感|null"
    }
  ],
  "structure_analysis": "整体结构一句话描述",
  "opening_analysis": "开头分析",
  "viral_points": ["爆点1", "爆点2"]
}

Step 5: Write to the Video Benchmark Library bitable

Field mapping:

bitable field	Value source
Source link	Original note URL
Platform	Xiaohongshu
Author	user.nickname
Publish date	time (timestamp → date format)
Play count	interactInfo view count (fill in if available)
Like count	interactInfo likedCount
Favorite count	interactInfo.collectedCount
Comment count	interactInfo.commentCount
Share count	interactInfo.shareCount
Body voiceover script	Concatenate all paragraph content in order
Voiceover script analysis	Aggregate structure_analysis + opening_analysis + viral_points
Content type	Determine based on the body text: knowledge takeaways/product recommendation voiceover/story narrative/opinion output/review comparison/tutorial sharing/trend commentary/other

Write API:

POST https://open.feishu.cn/open-apis/bitable/v1/apps/{VIDEO_BENCHMARK_APP_TOKEN}/tables/{VIDEO_BENCHMARK_TABLE_ID}/records

Step 6: Clean up temporary files

After completing the transcription and writing to bitable, explain the locations of the temporary video and transcription text, and ask the user whether to clean up. Only clean up after obtaining explicit permission, and prioritize moving to trash; do not directly rm -f。

---

Path 2: Image-text note → Social media viral topic library (default)

Target bitable: Social media viral topic library. AppToken and TableID are read from private configuration or environment variables, and are not hardcoded in the skill.

Step 1: Scrape the page and extract metadata

Prefer using the "one-click scraping script". Refer to this section only for manual handling.

Refer to "Core methods" to extract: title, desc, tagList, user.nickname, time, interactInfo

Step 2: Perform OCR/description on each image

For each image in imageList:

• Send to LLM for in-image text recognition + content summary
• Merged output: page-by-page summary + full-text restored version

Step 3: Write to the Social media viral topic library bitable

Field mapping:

bitable field	Value source
Social media viral topic library	Title (primary field)
Title	title
Source link	Original note URL
Platform	Xiaohongshu
Author	user.nickname
Date	time
One-sentence summary	LLM generates based on desc + image content
Article structure	LLM analyzes the image-text structure
Inspiration points	LLM extracts core insights
Content type	Determine based on the body text
Tags	Take the first 5 from tagList

Write API:

POST https://open.feishu.cn/open-apis/bitable/v1/apps/{SOCIAL_TOPIC_APP_TOKEN}/tables/{SOCIAL_TOPIC_TABLE_ID}/records

---

Path 2 (alternative): User-specified destination

If the user says "save to XXX library" or "add to XXX", create a record in the bitable specified by the user.

---

Platform identification

Domain	Platform
xiaohongshu.com	Xiaohongshu
douyin.com	Douyin
bilibili.com	Bilibili
weibo.com	Weibo
mp.weixin.qq.com	Official Account
zhihu.com	Zhihu
youtube.com	YouTube

---

Core scraping methods

URL and authentication context

• URL format:https://www.xiaohongshu.com/explore/{note_id}?xsec_token=...&xsec_source=pc_user
• Accessing HTML usually requires a valid login-state Cookie and xsec_token
• Login-state Cookies are sensitive credentials; do not write them into AGENTS, ordinary memory, public documents, or logs

Method A: Read webpage meta

curl -L -A 'Mozilla/5.0' '<笔记URL>'

Extract: og:title, description, og:image, og:video

Method B: Parse window.__INITIAL_STATE__(primary)

Key paths:root.note.noteDetailMap.<note_id>.note or note.noteDetailMap.<note_id>.note Prioritize checking: title, desc, type, tagList, user.nickname, interactInfo, video.media.stream, imageList For video direct links, prioritize checking:video.media.stream.h264[0].masterUrl

---

Decision rules

• type == "video" → video note, use pipeline one
• type == "normal" or imageList has content → image-text note, use pipeline two
• interactInfo all empty → may be a non-public note, mark "interaction data not public"

---

Failure handling

• Xiaohongshu risk control block → change User-Agent or add delay and retry
• Page opens but script reports no INITIAL_STATE → re-copy the full link with xsec_token ; if necessary, log in in the browser and then copy the link
• Video direct link 404 → try backupUrls
• Video direct link expired → rerun the one-click scraping script to refresh metadata.json and the direct link
• No subtitles → after downloading the video, call yichen-volc-asr for transcription
• yichen-volc-asr call failed → record the error, prompt the user that they can manually transcribe and then use the "body spoken script" field
• bitable write failed → print the full error response, prompt to check the token and table permissions

---

One-sentence workflow

Video note: scrape page → curl download video direct link → yichen-volc-asr transcription → LLM segmentation → write to video benchmark library → clean up temporary files with user permission Image-text note: scrape page → image OCR → LLM summary → write to social media topic library (default) or specified table