batch-research

Batch Research skill

This skill is used to guide the /weekly command on how to collect information efficiently, in batches, and concurrently from multiple data sources.

Core Responsibilities

As a batch scheduler, responsible for:

1. Parse the data source list
2. Generate all URLs to be crawled
3. Call concurrently in batches researcher agent
4. Aggregate results and generate a report

Workflow

Step 1: Preparation Phase

1. Read data sources: from .opencode/REFERENCE.md get the complete data source list
2. Generate URL list：
   • Static URLs: use directly
   • Dynamic URLs (Hacker News): use generateHNUrls(start_date, end_date)

import { generateHNUrls } from '.opencode/utils.mjs'

// Hacker News - 每天一个 URL
const hnUrls = generateHNUrls(start_date, end_date)
// 返回: [
//   "https://news.ycombinator.com/front?day=2026-03-22",
//   "https://news.ycombinator.com/front?day=2026-03-23",
//   ...
// ]

Step 2: Batching Strategy

Divide all URLs into 3 batches by priority, with 10-12 URLs per batch:

Batch	Data source type	Source
Batch 1	Important Resources	The "Important Resources" section in REFERENCE.md
Batch 2	Blogs & Websites	The "Blogs & Websites" section in REFERENCE.md
Batch 3	KOL & Influencers	The "KOL & Influencers" section in REFERENCE.md

Step 3: Concurrent Scheduling

Concurrency Configuration

Parameter	Value	Description
max_parallel	5	Maximum concurrency per round
batch_interval	3s	Wait time between batches
domain_rate_limit	2 req/domain	Same-domain rate limiting

Scheduling Rules

For each batch:

1. Execute in rounds: split the URLs in the batch by max_parallel into rounds (e.g., 12 URLs split into 3 rounds: 5 + 5 + 2)
2. Concurrency within a round: start all researcher agent
3. Wait for completion: wait for all researchers in the current round to return results
4. Proceed to the next round: after all in the current round are complete, wait batch_interval, then start the next round
5. After the batch is complete: proceed to the next batch

Parameter format for calling researcher：

url: https://news.ycombinator.com/front?day=2026-03-22
source_name: Hacker News
week_id: Y26W12
start_date: 2026-03-22
end_date: 2026-03-28
current_date: 2026-03-25
timezone: UTC+0

Concurrent call example(pseudocode):

# Batch 1: Important Resources
并行调用：
  - researcher(url: "https://news.ycombinator.com/front?day=2026-03-22", source_name: "Hacker News")
  - researcher(url: "https://news.ycombinator.com/front?day=2026-03-23", source_name: "Hacker News")
  - researcher(url: "https://drafts.miantiao.me/", source_name: "Miantiao Drafts")
  - researcher(url: "https://www.solidot.org/search?tid=151", source_name: "Solidot")
  - ...

等待 Batch 1 全部完成

# Batch 2: Blogs & Websites
并行调用：
  - researcher(url: "https://www.anthropic.com/engineering", source_name: "Anthropic Engineering")
  - researcher(url: "https://claude.com/blog", source_name: "Claude Blog")
  - ...

等待 Batch 2 全部完成

# Batch 3: KOL & Influencers
并行调用：
  - researcher(url: "https://baoyu.io/", source_name: "Baoyu")
  - ...

等待 Batch 3 全部完成

Step 4: Logging

All logs are written uniformly to logs/weekly-{week_id}.log，For human audit only, not used as a basis for recovery.

Log Format：

[2026-03-25T12:34:56Z] [PHASE1] [INFO] 开始抓取 Hacker News
[2026-03-25T12:35:10Z] [PHASE1] [OK] Hacker News - 5 篇文章
[2026-03-25T12:35:15Z] [PHASE1] [FAIL] daily.dev - 429 Too Many Requests (retried 2)

Log Level：

Level	Purpose
INFO	Phase/task start
OK	Task succeeded (including article count)
FAIL	Task failed (including error reason and retry count)

Step 5: Summary Report

After all batches are complete, generate a summary at the end of the log:

[2026-03-25T12:45:00Z] [PHASE1] [SUMMARY] 总数据源: 30 | 成功: 27 (X篇) | 失败: 3

Constraints and Notes

• Full crawl: must crawl all data sources, cannot skip
• Batch order: must execute in batch order, waiting for the current batch to complete before proceeding to the next batch
• Error isolation: a single researcher failure does not affect others
• Retries are handled by the researcher: this skill is not responsible for retries; the researcher handles them itself
• Before proceeding to the next phase: must complete crawling of all batches