X (Twitter) tweet scraping Skill, supporting single tweets and user timelines, with automatic screenshot saving and JSON export.
Features
- ✅ Supports scraping single tweets
- ✅ Supports scraping user timelines
- ✅ Automatically extracts tweet text, author, and timestamp
- ✅ Supports longform tweets
- ✅ Saves full-page screenshots
- ✅ Exports results in JSON format
- ✅ Supports virtual environments (e.g., playwright-env)
- ✅ Headless mode (UI-less server deployment)
- ✅ Command-line tool support
- ✅ Direct interface invocation (no need to manually run scripts)
Quick Start
1. Use the command-line tool
# 爬取单条推文
python3 skills/x-tweet-fetcher/scripts/fetch.py \
https://x.com/heynavtoor/status/2028148844891152554
# 爬取用户推文
python3 skills/x-tweet-fetcher/scripts/fetch.py \
--user elonmusk --max-tweets 20
# 指定输出目录
python3 skills/x-tweet-fetcher/scripts/fetch.py \
https://x.com/heynavtoor/status/2028148844891152554 \
--output ./x_tweets
2. Run using a virtual environment
# 方式 1: 激活虚拟环境后运行
source playwright-env/bin/activate
python3 skills/x-tweet-fetcher/scripts/fetch.py <url>
# 方式 2: 使用提供的包装脚本
python3 skills/x-tweet-fetcher/scripts/run_in_venv.py \
<url> --venv /path/to/playwright-env
3. Direct invocation (recommended)
Simply tell the assistant the tweet URL, and it will automatically execute and return the result:
帮我爬取这篇推文：https://x.com/heynavtoor/status/2028148844891152554
4. Use in code
from x_tweet_fetcher import fetch_tweet, fetch_user_tweets
# 爬取单条推文
result = fetch_tweet("https://x.com/heynavtoor/status/2028148844891152554")
print(f"作者: {result['author']}")
print(f"时间: {result['timestamp']}")
print(f"内容: {result['text'][:500]}")
# 爬取用户推文
results = fetch_user_tweets("elonmusk", max_tweets=20)
for tweet in results['tweets']:
print(f"{tweet['author']}: {tweet['text'][:100]}")
Command-line tools
fetch.py - Main scraping tool
python3 skills/x-tweet-fetcher/scripts/fetch.py [OPTIONS] [URL]
参数:
URL X 推文 URL（与 --user 二选一）
选项:
-u, --user TEXT 用户名（与 URL 二选一）
-m, --max-tweets INTEGER 最大推文数量（默认: 20）
-o, --output PATH 输出目录（默认: ./x_tweets）
--no-screenshot 不保存截图
--headless BOOLEAN 无头模式（默认: true）
--timeout INTEGER 超时时间（毫秒，默认: 120000）
-h, --help 显示帮助信息
fetch_direct.py - Direct invocation script
# 直接调用（已配置好虚拟环境）
python3 skills/x-tweet-fetcher/fetch_direct.py <tweet_url>
run_in_venv.py - Virtual environment run tool
python3 skills/x-tweet-fetcher/scripts/run_in_venv.py \
<url> --venv /path/to/playwright-env
Output Description
Single Tweet
Each scrape creates a directory named with a timestamp:
x_tweets/
└── 20260302_142000/
├── tweet.json # JSON 格式结果
└── tweet.png # 完整页面截图
tweet.json structure
{
"url": "https://x.com/heynavtoor/status/2028148844891152554",
"fetch_time": "2026-03-02T14:20:00.000000",
"tweet": {
"tweet_id": "2026-03-01T16:42:20.000Z_1234567890",
"text": "推文内容...",
"full_text": "完整推文内容（包含作者信息等）...",
"author": "Nav Toor @heynavtoor",
"timestamp": "2026-03-01T16:42:20.000Z",
"extracted_at": "2026-03-02T14:20:00.000000"
}
}
User Timeline
x_tweets/
└── elonmusk_20260302_142000/
├── tweets.json # JSON 格式结果
└── timeline.png # 时间线截图
tweets.json structure
{
"username": "elonmusk",
"url": "https://x.com/elonmusk",
"fetch_time": "2026-03-02T14:20:00.000000",
"tweets_collected": 20,
"tweets": [
{
"tweet_id": "...",
"text": "...",
"author": "...",
"timestamp": "...",
"extracted_at": "..."
}
]
}
Configuration File
You can create config.json to customize the default configuration:
{
"headless": true,
"timeout": 120000,
"output_dir": "./x_tweets",
"save_screenshot": true,
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36..."
}
Usage Examples
Example 1: Scrape a single tweet
python3 skills/x-tweet-fetcher/scripts/fetch.py \
https://x.com/heynavtoor/status/2028148844891152554
Example 2: Scrape a user's tweets
python3 skills/x-tweet-fetcher/scripts/fetch.py \
--user elonmusk --max-tweets 20
Example 3: Direct invocation (simplest)
Simply tell the assistant:
帮我爬取这篇推文：https://x.com/heynavtoor/status/2028148844891152554
The assistant will automatically execute and return a summary of the results!
Example 4: Use in a virtual environment
python3 skills/x-tweet-fetcher/scripts/run_in_venv.py \
https://x.com/heynavtoor/status/2028148844891152554 \
--venv /opt/playwright-env
Best Practices
- Use a virtual environment: Isolate dependencies to avoid conflicts
- Set a reasonable timeout: X pages may load slowly; 120 seconds is recommended
- Control scraping frequency: Avoid account restrictions
- Use headless mode: Use when deploying on a server
headless: true - Regularly check the output: Ensure the content is complete
Notes
⚠️ Important Notes：
- X has strong anti-scraping mechanisms; please use responsibly
- You may need to log in to an account to get complete content
- It is recommended to add accounts using cookies (refer to twscrape)
- Comply with X's Terms of Service; use only for personal learning and research
Troubleshooting
Issue: Module not found
Solution: Make sure you are running in the correct virtual environment, or install the dependencies:
pip install playwright
playwright install chromium
Issue: Browser fails to start
Solution: Install system dependencies:
sudo apt install -y libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 \
libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 \
libxfixes3 libxrandr2 libgbm1 libasound2
Issue: Unable to extract tweet text
Solution:
- Check whether the page has fully loaded
- Try increasing the timeout
- The page structure may have changed; the selectors need to be updated
Issue: Page load timeout
Solution: Increase the timeout:
python3 skills/x-tweet-fetcher/scripts/fetch.py \
<url> --timeout 180000
Related Resources
- Playwright Official Documentation
- X Developer Platform
- twscrape - A more powerful X API scraper
Changelog
v1.0.0
- Initial version
- Supports scraping a single tweet
- Supports scraping user timelines
- Supports longform tweets
- Supports JSON export
- Supports full-page screenshots
- Supports virtual environments
- Provides a command-line tool
- Provides a direct call interface