SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsxianyu-data-grabber

xianyu-data-grabber

Data & analytics

View repository

Updated 10-16 03:10

About this skill

Xianyu Data Scraping Skill (xianyu-data-grabber)

Feature Description

Uses Playwright + OCR technology to bypass Xianyu's anti-scraping, scrape product data, and automatically upload it to a Gitee repository.

Core Capabilities:

  • Batch keyword search (supports 15+ keywords)
  • Automatic screenshot saving (PNG format)
  • OCR text recognition (Chinese + English)
  • Product information extraction (title, price, number of interested users)
  • Automatic analysis report generation (Markdown + JSON)
  • Automatic upload to Gitee repository

When to Use

Activate this skill when the user mentions the following scenarios:

  1. "Help me scrape competitor data on Xianyu"
  2. "Research products in a certain category on Xianyu"
  3. "Analyze the pricing strategies of top Xianyu sellers"
  4. "Xianyu market research"
  5. "Scrape Xianyu product prices and sales"
  6. "xianyu research" / "Xianyu data scraping"
  7. "See how well this thing sells on Xianyu"
  8. "Xianyu competitor analysis"

Configuration File

1. Basic Configuration

Create ~/.openclaw/workspace/.xianyu-grabber-config.json:

{
  "gitee": {
    "token": "your_gitee_token",
    "owner": "your_username",
    "repo": "xianyu-data"
  },
  "xianyu": {
    "cookie": "your_xianyu_cookie"
  },
  "grabber": {
    "keywords": ["Magisk", "KernelSU", "手机维修"],
    "screenshotDir": "legion/screenshots",
    "dataDir": "legion/data",
    "uploadToGitee": true,
    "ocrLanguage": "chi_sim+eng"
  }
}

2. Gitee Personal Access Token

How to obtain:

  1. Log in to https://gitee.com
  2. Settings → Personal Access Tokens
  3. Create a new token (check the projects permission)
  4. Copy the token to the configuration file

3. Xianyu Cookie (optional)

How to obtain:

  1. Log in to Xianyu in the browser
  2. F12 Developer Tools → Network
  3. Refresh the page → copy the Cookie field

Usage

Basic Usage

# 抓取单个关键词
xianyu-data-grabber search "Magisk"

# 抓取多个关键词
xianyu-data-grabber search "Magisk" "KernelSU" "root"

# 使用配置文件中的关键词列表
xianyu-data-grabber search --config

Advanced Usage

# 指定输出目录
xianyu-data-grabber search "Magisk" --output ./my-research

# 不上传 Gitee
xianyu-data-grabber search "Magisk" --no-upload

# 仅 OCR 不抓取（已有截图）
xianyu-data-grabber ocr --input ./screenshots

# 生成汇总报告
xianyu-data-grabber report --input ./data

# 上传到 Gitee
xianyu-data-grabber upload --all

Invoke via the messaging skill

帮我抓取闲鱼上"Magisk"相关的商品数据
调研闲鱼手机维修类目的竞品
分析闲鱼 root 服务的定价策略

Output Files

Screenshot Files

  • legion/screenshots/xianyu-{keyword}.png
  • Full-page screenshot (height may exceed 10000px)

Data Files

File Format Content
xianyu-{keyword}.json JSON Raw data for a single keyword
xianyu-full-data.json JSON Summary of all keywords
xianyu-summary.md Markdown Summary report
xianyu-analysis.md Markdown In-depth analysis report

Gitee Repository Structure

xianyu-data/
├── README.md              # 自动生成的说明
├── data/
│   ├── xianyu-full-data.json
│   └── xianyu-{keyword}.json
├── screenshots/
│   └── xianyu-{keyword}.png
├── reports/
│   ├── xianyu-summary.md
│   └── xianyu-analysis.md
└── upload-{timestamp}.md  # 上传记录

Core Scripts

grabber.js - Main scraping script

// 1. 启动 Playwright 浏览器（Headless + 伪装）
// 2. 加载 Cookie（如有）
// 3. 遍历关键词搜索
// 4. 截图保存
// 5. 调用 OCR 识别
// 6. 提取商品信息
// 7. 保存 JSON 数据

ocr.py - OCR recognition script

# 1. 读取截图
# 2. Tesseract OCR 识别
# 3. 提取价格/想要人数等
# 4. 输出结构化数据

uploader.sh - Gitee upload script

# 1. 调用 Gitee API
# 2. 创建/更新文件
# 3. 提交 commit
# 4. 返回上传结果

Data Format

Single Product Data

{
  "keyword": "Magisk",
  "products": [
    {
      "title": "Magisk 模块合集 17G 资源",
      "price": "1.00",
      "wants": "628 人想要",
      "seller": "卖家信用优秀",
      "tags": ["24h 自动发货", "包邮"]
    }
  ],
  "timestamp": "2026-03-20T06:00:00+08:00",
  "screenshot": "screenshots/xianyu-Magisk.png"
}

Summary Report Structure

# 闲鱼数据调研报告

## 关键词：Magisk
- 商品数：19 个
- 价格区间：¥1-50 元
- 热门商品：...

## 关键词：KernelSU
...

## 价格分析
...

## 竞品分析
...

DEBUG Guide

Log Locations

Log File
Scraping log logs/xianyu-grabber.log
OCR log logs/xianyu-ocr.log
Upload log logs/xianyu-upload.log
Error log logs/xianyu-error.log

Common Issues

1. Screenshot shows "Illegal Access"

Cause: Anti-scraping detected automation

Solution:

# 1. 更新 Cookie
# 2. 降低抓取速度（增加延迟）
# 3. 减少并发关键词数量

2. OCR recognition result is empty

Cause: Tesseract is not installed or the language pack is missing

Solution:

# 安装 Tesseract
apt-get install tesseract-ocr tesseract-ocr-chi-sim

# 验证安装
tesseract --version
tesseract --list-langs

3. Gitee upload failed

Cause: Invalid token or insufficient permissions

Solution:

# 1. 检查 Token 是否有效
curl -H "Authorization: Bearer YOUR_TOKEN" https://gitee.com/api/v5/user

# 2. 检查仓库权限
# 确保 Token 有 projects 权限

4. Playwright browser failed to start

Cause: Missing dependencies or browser not installed

Solution:

# 安装 Playwright 浏览器
npx playwright install chromium

# 安装系统依赖
apt-get install libnss3 libnspr4 libatk1.0-0 \
  libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 \
  libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
  libgbm1 libasound2 libpango-1.0-0 libcairo2

Test command

# 测试配置
xianyu-data-grabber test-config

# 测试 OCR
xianyu-data-grabber test-ocr --image test.png

# 测试 Gitee 上传
xianyu-data-grabber test-upload --file test.txt

# 完整测试
xianyu-data-grabber test --all

Scheduled tasks

Daily automatic scraping

# crontab -e
0 9 * * * cd ~/.openclaw/workspace && node skills/xianyu-data-grabber/grabber.js --config --upload >> logs/xianyu-cron.log 2>&1

Generate reports weekly

0 10 * * 1 cd ~/.openclaw/workspace && node skills/xianyu-data-grabber/report.js >> logs/xianyu-report.log 2>&1

Security and privacy

Sensitive data protection

  • Cookie: Stored in configuration file, permissions 600
  • Gitee Token: Stored in configuration file, permissions 600
  • Data files: Stored locally, not uploaded to third parties

Platform compliance

  • Request frequency: Default 5-second interval/keyword
  • User-Agent: Real browser identifier
  • Data usage: For personal research only

Performance optimization

Batch scraping

# 并行抓取（更快但可能被检测）
xianyu-data-grabber search --parallel 3

# 串行抓取（更慢但更安全）
xianyu-data-grabber search --sequential

Caching mechanism

  • Screenshot cache: avoid repeated scraping
  • OCR cache: avoid repeated recognition
  • Data cache: 5-minute validity period

Related files

  • Skill file:skills/xianyu-data-grabber/SKILL.md
  • Main script:skills/xianyu-data-grabber/grabber.js
  • OCR script:skills/xianyu-data-grabber/ocr.py
  • Upload script:skills/xianyu-data-grabber/uploader.sh
  • Configuration file:.xianyu-grabber-config.json

Changelog

v1.0.0 (2026-03-20)

  • 🎉 Initial version
  • Playwright + OCR scraping
  • Gitee automatic upload
  • Batch keyword support
  • Automatically generate reports