xiaohongshu-crawler

Xiaohongshu post scraping (Dubaoya)

Quack! Give this duck a keyword, and this duck will scrape back the popular posts in that track on Xiaohongshu—you can also set a date range, choose the sorting criterion, and the terminal will directly lay out a clickable post table, plus a line of topic insight.

Data goes through doubaoya.com in one line; authentication uses your own key (environment variable DOUBAOYA_API_KEY, in the form dyh_…）。

Applicable scenarios

Scenario	How to use	What you get
Track viral post review	Search `减脂餐` to see recent popular posts in this track	Grasp the current viral angles at a glance
Research for a specified time period	Search `露营` to set the June period	Locate popular content in a specific time period
Find benchmarks by popularity	Search `通勤穿搭` use `--sort-type _4` to pull the hottest	Directly benchmark against top posts
Track the latest updates	Search `多巴胺穿搭` use `--sort-type _2` to pull the latest	Understand the latest trends in the track
Topic/competitor analysis	Search track words to see how everyone writes titles and approaches angles	Find direction for the next post

Workflow (4 steps)

1. Understand the intent, extract keywords and filter conditions

Keyword: extract from the user's description thecore track word. The shorter and broader the word, the more hits; too long and too specific easily returns nothing. Search only one keyword at a time.
Date range(optional): if the user says "recent/June/last week", etc. → parse into --start-date/--end-date（YYYY-MM-DD). If no time is mentioned,do not passthese two parameters; leave it to the server default handling.
Sorting(optional, default comprehensive): if the user says "hottest/most popular" → _4; "latest/by time" → _2; if not mentioned, use the default _0(comprehensive/relevant).

2. Call the scraping script

Basic (only keyword, default date, comprehensive sorting):

python3 "$SKILL_PATH/scripts/crawl_xhs.py" "减脂餐"

With date range + sorting:

python3 "$SKILL_PATH/scripts/crawl_xhs.py" "露营" --start-date 2026-06-01 --end-date 2026-06-20 --sort-type _4

The script prints the data from the success envelope to stdout as JSON.Run the script only once per query, read the complete stdout directly, do not use head/tail preview or repeatedly call for the same keyword.

Sorting criteria (sortType)：_0 comprehensive/relevant (default),_2 latest (by publish time),_4 hottest (by interaction count).

3. Render the post table

From data.items(work array), read the fields and lay them out as a Markdown table in the format below. Read the fields defensively—title(title),authorNickname(author),likedCount(likes), and the publish time may be missing; if it's missing, leave it blank or write "—", don't throw an error. Render the title as aclickable link(pointing to the original work URL, if any). If the like count is ≥ 10000, use the x.xw format.

Title	Author	Likes	Publish Time
Fat-loss meal recipes to lose 5 jin in 3 days	Fat-Loss Diary	5.6w	2026-06-18
Camping gear checklist done in one article	Mountain-Style Life	2.3w	2026-06-15

When there are many items, you can first show the first several and prompt, "X total, first N shown, want to see all?"

4. Give one line of topic insight

After the table, add one line in this duck's tone—Topic Insight: what angle these works are competing for, which opening no one has written thoroughly yet, and whether it's worth following. Keep it short and useful, don't pile on clichés, don't ask the user "what is your real purpose", and don't make up data.

Get the key (secret key)

Open doubaoya.com
Log in
and go to Key Center
Generate Key(in the form dyh_…）

Configure it as an environment variable (the script only recognizes this):

export DOUBAOYA_API_KEY="dyh_你的密钥"

Iron rule: never print the key, never write it to a file, never echo it back to the user. The script itself also never outputs the key. All requests are sent only to doubaoya.com, don't take the key to any other domain.

API and Envelope

POST https://doubaoya.com/api/apis/xiaohongshu/crawl-work/call
Auth header:Authorization: Bearer $DOUBAOYA_API_KEY
Request body:
```
{ "keyword": "减脂餐", "sortType": "_0" }
```
- keyword: string, required
- sortType: string, has a default value _0(comprehensive/related), the scriptalways includes it; optional _2(latest),_4(hottest)
- startDate / endDate: string YYYY-MM-DD, optional—only include it when the user gives a time range, otherwise omit it and let the server use the default

Return envelope:

{
  "success": true,
  "requestId": "...",
  "data": { "items": [ { "title": "…", "authorNickname": "…", "likedCount": 56000 } ] },
  "error": null
}

data.items: work list (each item contains title title,authorNickname author,likedCount likes, and publish time/link, etc.)

First check success: only read true if it is data; otherwise read error.code / error.message。

Error Handling

When the script fails, it prints to stderr [error] CODE: message and exits with code 1. Common cases:

HTTP	code	Meaning	Handling
401	`MISSING_API_KEY` / `UNAUTHORIZED`	No key provided or invalid key	Check `DOUBAOYA_API_KEY`, go to Key Center to regenerate.Do not echo the key
400	`VALIDATION_ERROR`	Invalid parameters (e.g. empty keyword, wrong date format)	Fix the keyword/date format and retry
402	`INSUFFICIENT_CREDITS`	Insufficient quota	Go to doubaoya.com to top up/renew quota
404	`ENDPOINT_NOT_FOUND`	Wrong API path	Usually the script has been modified; restore the default endpoint path
502	`PROVIDER_FAILED`	Temporary upstream failure (already automatically refunded）	Safe to retry
—	`NETWORK_ERROR`	Cannot connect to doubaoya.com	Check your network and try again

502 PROVIDER_FAILED A refund will be issued automatically; retrying is safe and will not result in duplicate charges. When 0 results are found, it is recommended to use shorter/broader keywords or widen the time range; do not fabricate data.

Directory structure

xiaohongshu-crawler/
├── SKILL.md             # 本文件
└── scripts/
    └── crawl_xhs.py     # 零依赖爬取脚本（urllib），调用 doubaoya.com

About the notice field in the response

After calling the doubaoya API corresponding to this skill, if the following appears at the top level of the returned JSON: notice field, please relay this sentence to the user verbatim—this is a notice that this skill has an update, it does not affect the result of this call, and there is no need to retry or change behavior.