twitter-crawler

Twitter Tweet Crawler

Overview

Crawl tweets from a specified Twitter user and save them in Markdown format.

Usage

Basic Usage

cd ~/.claude/skills/twitter-crawler
python3 scripts/fetch_tweets.py <用户名>

Full Parameters

python3 scripts/fetch_tweets.py <用户名> [选项]

Options: -n, --limit N Number of tweets to fetch (default 10) -p, --pages N Number of pages to fetch (default 1, about 20 per page) -o, --output PATH Output file path (default outputs/{username}.md) --json Also output JSON format --interval SECONDS Request interval in seconds (default 5, to prevent rate limiting) --no-user-info Do not fetch user info --title TITLE Custom Markdown title --fields FIELDS Fields to include, comma-separated --auth-token TOKEN Specify auth_token

### 频率限制

Twitter API 有频率限制，脚本内置了以下保护机制：
- 默认每次请求间隔 5 秒
- 自动添加随机抖动（0.5-2秒）避免被检测
- 可通过 `--interval` 调整间隔时间
- 获取超过 20 条推文时自动分页

### 字段选项

`--fields` 可选值（逗号分隔）：
- `text` - 推文内容
- `time` - 发布时间
- `likes` - 点赞数
- `retweets` - 转发数
- `replies` - 回复数
- `views` - 浏览量
- `url` - 推文链接
- `media` - 媒体（图片）

## 示例

### 示例1：基础爬取

```bash
python3 scripts/fetch_tweets.py VoxcatAI

Output:outputs/VoxcatAI.md

Example 2: Specify count and output path

python3 scripts/fetch_tweets.py elonmusk -n 20 -o ~/Desktop/elon.md

Example 3: Only content and links

python3 scripts/fetch_tweets.py sama --fields text,url

Example 4: Also output JSON

python3 scripts/fetch_tweets.py OpenAI -n 15 --json

Output:

• outputs/OpenAI.md
• outputs/OpenAI.json

Example 5: Custom title

python3 scripts/fetch_tweets.py VoxcatAI --title "VoxCat 的 Prompt 分享"

Output Format

Markdown Format

# @用户名 的推文

> 爬取时间: 2026-01-11 12:00:00
> 推文数量: 10

## 用户信息

- **名称**: xxx
- **用户名**: @xxx
- **粉丝**: 1,000
- **简介**: xxx

## 推文列表

### 1. 推文

**时间**: 2026-01-11 10:00:00

> 推文内容...

**互动**: ❤️ 100 | 🔁 20 | 💬 5

**链接**: [https://twitter.com/xxx/status/xxx](...)

---

JSON Format

{
  "user": {
    "name": "xxx",
    "username": "xxx",
    "followers": 1000,
    ...
  },
  "tweets": [
    {
      "id": "xxx",
      "text": "xxx",
      "created_at": "2026-01-11 10:00:00",
      "likes": 100,
      "retweets": 20,
      ...
    }
  ],
  "fetched_at": "2026-01-11T12:00:00"
}

Configuration

auth_token

The script will automatically read from ~/Documents/trend-crawler-master/trend-crawler/config.yaml read auth_token。

If separate configuration is needed, create ~/.claude/skills/twitter-crawler/config.yaml：

twitter_accounts:
  auth_token: "你的auth_token"

Get auth_token

1. Log in to the Twitter web version
2. Open developer tools (F12) → Application → Cookies
3. Find auth_token value

Dependencies

• tweety-ns (loaded from the trend-crawler project's venv)
• pyyaml

Notes

1. Rate limiting: Twitter has API rate limits, it is recommended to use intervals
2. auth_token expiration: If errors occur, you may need to update auth_token
3. Guest mode: It can be used without configuring auth_token, but may be limited