xiaohongshu-ingest

Xiaohongshu Collection + Viral Breakdown Skill

Scrape Xiaohongshu notes into structured Markdown for ingestion, then use DeepSeek to break down the viral logic and derive topic ideas that can be written directly. Automatically distinguishes image-text from video notes: image-text → download images and embed locally; video → convert to a text transcript just like Douyin. This fills the repository's biggest platform gap in "Chinese content collection" and is also the starting point of the "collection → topic selection" creative workflow.

Environment Requirements

# 图文采集零依赖（仅 Python 标准库）

# 视频笔记转录需要（与抖音/B站转录同一套依赖）：
pip install funasr modelscope torch torchaudio
# macOS: brew install ffmpeg   |   Ubuntu: sudo apt install ffmpeg

# 强烈建议：配置 cookie 以规避风控（从浏览器登录后复制）
export XHS_COOKIE="你的小红书 cookie 字符串"

Viral breakdown requires a DeepSeek API Key

export DEEPSEEK_API_KEY="your-api-key"


## 使用方法

```bash
# 1) 采集笔记 → 统一 frontmatter Markdown
python scripts/fetch_note.py "https://www.xiaohongshu.com/explore/xxxx" -o ./out
python scripts/fetch_note.py "http://xhslink.com/xxxx" -o ./out
python scripts/fetch_note.py "链接" -o ./out --no-images   # 图文：只留图片链接不下载
python scripts/fetch_note.py "链接" -o ./out --no-video    # 视频：不转录，只留视频链接
python scripts/fetch_note.py "链接" -o ./out --fallback-text 手动正文.txt

# 2) 爆款拆解 → 拆解报告 + 5 条衍生选题
python scripts/analyze_hook.py ./out/某笔记.md -o ./out

Output

fetch_note.py: title, body text, tags, author, engagement data (likes/saves/comments) → unified frontmatter Markdown (platform: xiaohongshu, including note_type: image|video）。Routing by note type：
- Image-text notes: images downloaded locally to <标题>.assets/ and embedded with ![]() (the body text is often in the images; after localization it is directly visible in Obsidian).--no-images only keeps the link; if a single image fails, it automatically falls back to a link
- Video notes: extract the direct video link → extract audio with ffmpeg → transcribe with SenseVoice into a text transcript written to ## 视频文字稿。--no-video only keeps the video link; if funasr/ffmpeg is missing, it automatically degrades to saving the link
analyze_hook.py: target audience / scenario / pain point / emotional value / title hook / body structure / reusable template / 5 derivative topic ideas (with - [ ] checkboxes, can go directly into the topic library)

⚠️ About anti-scraping (must read first)

Xiaohongshu has strict risk control;pure script access without login is often blocked. This skill's strategy:

Prioritize parsing the __INITIAL_STATE__ structured data embedded in the page
If that fails, fall back to the og: meta tags
If it still fails, give a clear prompt—at that point you canmanually copy the note body textand save it as .txt, use --fallback-text to continue generating unified Markdown, or go directly to analyze_hook.py for breakdown (the breakdown step does not depend on scraping)

Providing XHS_COOKIE can significantly improve the collection success rate. If the page structure changes, you need to update the selectors in fetch_note.py .

Compliance Statement

Forpersonal learning and researchonly. Please comply with Xiaohongshu's user agreement and robots rules, control request frequency, and do not use it for bulk scraping, commercial crawling, or scenarios that infringe on the rights of others.

Connecting the Workflow

Collection output → knowledge-base-management ingestion (unified frontmatter, aggregated by platform )
Derivative topic ideas → creator topic flow (collection → breakdown → topic library → draft)
Multiple breakdowns → build up a "viral topic library", connecting to industry-intelligence-radar trend signals

References

Xiaohongshu viral methodology: audience × scenario × pain point + emotional value
DeepSeek — LLM used for breakdown