Based on MediaCrawler multi-platform public information collection tool.
Supported Platforms
- Xiaohongshu (xhs)
- Douyin (dy)
- Kuaishou (ks)
- Bilibili (bili)
- Weibo (wb)
- Tieba (tieba)
- Zhihu (zhihu)
Features
- Automatic dependency installation
- Keyword search collection
- Collection by specified post/content ID
- Creator homepage collection
- Comment/sub-comment scraping
- Login state caching
- WebUI visual operation
- Multiple data storage options (CSV, JSON, JSONL, Excel, SQLite, MySQL, MongoDB)
- Quick location of result files
Usage
Install Environment
bash scripts/setup.sh
View Help
cd "$PROJECT_PATH"
uv run main.py --help
Run Collection
Xiaohongshu - Keyword Search
uv run main.py --platform xhs --lt qrcode --type search --keywords "护肤" --headless false
Douyin - Keyword Search
uv run main.py --platform dy --lt qrcode --type search --keywords "护肤" --headless false
Specified Post Detail Scraping
uv run main.py --platform xhs --lt qrcode --type detail --specified_id "帖子ID1,帖子ID2"
Creator Homepage Scraping
uv run main.py --platform xhs --lt qrcode --type creator --creator_id "创作者ID1"
Start WebUI
uv run uvicorn api.main:app --port 8080 --reload
After starting, visit:
http://127.0.0.1:8080
Data Storage
According to config/base_config.py in:
SAVE_DATA_OPTION = "jsonl"
SAVE_DATA_PATH = ""
By default, results are saved to:
data/{平台}/{存储格式}/
For example, Douyin JSONL:
data/douyin/jsonl/search_contents_YYYY-MM-DD.jsonl
data/douyin/jsonl/search_comments_YYYY-MM-DD.jsonl
data/douyin/jsonl/search_creators_YYYY-MM-DD.jsonl
For example, Xiaohongshu JSONL:
data/xiaohongshu/jsonl/search_contents_YYYY-MM-DD.jsonl
data/xiaohongshu/jsonl/search_comments_YYYY-MM-DD.jsonl
If you set:
--save_data_path "/your/custom/path"
then the results will be written to your specified directory.
Quickly View Results
bash scripts/show_results.sh
This script will list the result files in the data/ directory under the current project.
Prerequisites
- Git
- uv (the script can install it automatically)
- Playwright browser driver (the script automatically installs Chromium)
Notes
- For learning and research use only
- Prohibited for illegal purposes or infringing on the legitimate rights and interests of others
- Prohibited for commercial non-compliant scraping
- Before execution, you should confirm that the target behavior is legal and compliant