Architecture Upgrade: Main Agent orchestration + SubAgent execution + browser scraping + intelligent caching
Core Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ 主 Agent (Orchestrator) │
│ 职责：调度、监控、评估、决策、汇总 │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ 1. 初始化 │ → │ 2. 调度 │ → │ 3. 监控 │ → │ 4. 评估 │ │
│ │ 读取配置 │ │ 分发任务 │ │ 收集结果 │ │ 筛选排序 │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ 5. 决策 │ ← │ 够20条？ │ │ 6. 生成 │ → │ 7. 更新 │ │
│ │ 继续/停止 │ │ Y/N │ │ 日报文件 │ │ 缓存统计 │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────┘
↓ 调度 ↑ 返回结果
┌─────────────────────────────────────────────────────────────────────┐
│ SubAgent 执行层 │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Worker A │ │ Worker B │ │ Browser │ │
│ │ (WebFetch) │ │ (WebFetch) │ │ (Headless) │ │
│ │ Tier1 Batch │ │ Tier2 Batch │ │ JS渲染页面 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ ↓ ↓ ↓ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 结构化结果返回 │ │
│ │ { status, data: [...], errors: [...], metadata: {...} } │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Configuration Files
This Skill uses the following configuration files:
| File | Purpose |
|---|---|
sources.json |
Information source configuration, priority, scraping method |
cache.json |
Cached data, historical statistics, deduplication fingerprints |
Detailed Execution Workflow
Phase 1: Initialization
步骤:
1. 确定日期（用户参数或当前日期）
2. 读取 sources.json 获取源配置
3. 读取 cache.json 获取历史数据
4. 创建输出目录 NewsReport/
5. 检查今日是否已有部分报告（追加模式）
Phase 2: Dispatch SubAgents
Strategy: Parallel scheduling, batched execution, early stopping mechanism
第1波 (并行):
- Worker A: Tier1 Batch A (HN, HuggingFace Papers)
- Worker B: Tier1 Batch B (OneUsefulThing, Paul Graham)
等待结果 → 评估数量
如果 < 15 条高质量:
第2波 (并行):
- Worker C: Tier2 Batch A (James Clear, FS Blog)
- Worker D: Tier2 Batch B (HackerNoon, Scott Young)
如果仍 < 20 条:
第3波 (浏览器):
- Browser Worker: ProductHunt, Latent Space (需要JS渲染)
Phase 3: SubAgent Task Format
The task format each SubAgent receives:
task: fetch_and_extract
sources:
- id: hn
url: https://news.ycombinator.com
extract: top_10
- id: hf_papers
url: https://huggingface.co/papers
extract: top_voted
output_schema:
items:
- source_id: string # 来源标识
title: string # 标题
summary: string # 2-4句摘要
key_points: string[] # 最多3个要点
url: string # 原文链接
keywords: string[] # 关键词
quality_score: 1-5 # 质量评分
constraints:
filter: "前沿技术/高深技术/提效技术/实用资讯"
exclude: "泛科普/营销软文/过度学术化/招聘帖"
max_items_per_source: 10
skip_on_error: true
return_format: JSON
Phase 4: Main Agent Monitoring and Feedback
Main Agent responsibilities:
监控:
- 检查 SubAgent 返回状态 (success/partial/failed)
- 统计收集到的条目数量
- 记录每个源的成功率
反馈循环:
- 如果某 SubAgent 失败，决定是否重试或跳过
- 如果某源持续失败，标记为禁用
- 动态调整后续批次的源选择
决策:
- 条目数 >= 25 且高质量 >= 20 → 停止抓取
- 条目数 < 15 → 继续下一批
- 所有批次完成但 < 20 → 用现有内容生成（宁缺毋滥）
Phase 5: Evaluation and Filtering
去重:
- 基于 URL 完全匹配
- 基于标题相似度 (>80% 视为重复)
- 检查 cache.json 避免与历史重复
评分校准:
- 统一各 SubAgent 的评分标准
- 根据来源可信度调整权重
- 手动标注的高质量源加分
排序:
- 按 quality_score 降序
- 同分按来源优先级排序
- 截取 Top 20
Phase 6: Browser Scraping (MCP Chrome DevTools)
For pages that require JS rendering, use a headless browser:
流程:
1. 调用 mcp__chrome-devtools__new_page 打开页面
2. 调用 mcp__chrome-devtools__wait_for 等待内容加载
3. 调用 mcp__chrome-devtools__take_snapshot 获取页面结构
4. 解析 snapshot 提取所需内容
5. 调用 mcp__chrome-devtools__close_page 关闭页面
适用场景:
- ProductHunt (403 on WebFetch)
- Latent Space (Substack JS 渲染)
- 其他 SPA 应用
Phase 7: Generate Daily Report
输出:
- 目录: NewsReport/
- 文件名: YYYY-MM-DD-news-report.md
- 格式: 标准 Markdown
内容结构:
- 标题 + 日期
- 统计摘要（源数量、收录数量）
- 20条高质量内容（按模板）
- 生成信息（版本、时间戳）
Phase 8: Update Cache
更新 cache.json:
- last_run: 记录本次运行信息
- source_stats: 更新各源统计数据
- url_cache: 添加已处理的 URL
- content_hashes: 添加内容指纹
- article_history: 记录收录文章
SubAgent Invocation Examples
Use the general-purpose Agent
Since custom agents require a session restart to be discovered, you can use general-purpose and inject the worker prompt:
Task 调用:
subagent_type: general-purpose
model: haiku
prompt: |
你是一个无状态的执行单元。只做被分配的任务，返回结构化 JSON。
任务：抓取以下 URL 并提取内容
URLs:
- https://news.ycombinator.com (提取 Top 10)
- https://huggingface.co/papers (提取高投票论文)
输出格式：
{
"status": "success" | "partial" | "failed",
"data": [
{
"source_id": "hn",
"title": "...",
"summary": "...",
"key_points": ["...", "...", "..."],
"url": "...",
"keywords": ["...", "..."],
"quality_score": 4
}
],
"errors": [],
"metadata": { "processed": 2, "failed": 0 }
}
筛选标准：
- 保留：前沿技术/高深技术/提效技术/实用资讯
- 排除：泛科普/营销软文/过度学术化/招聘帖
直接返回 JSON，不要解释。
Use the worker Agent (requires session restart)
Task 调用:
subagent_type: worker
prompt: |
task: fetch_and_extract
input:
urls:
- https://news.ycombinator.com
- https://huggingface.co/papers
output_schema:
- source_id: string
- title: string
- summary: string
- key_points: string[]
- url: string
- keywords: string[]
- quality_score: 1-5
constraints:
filter: 前沿技术/高深技术/提效技术/实用资讯
exclude: 泛科普/营销软文/过度学术化
Output Template
# Daily News Report（YYYY-MM-DD）
> 本日筛选自 N 个信息源，共收录 20 条高质量内容
> 生成耗时: X 分钟 | 版本: v3.0
>
> **Warning**: Sub-agent 'worker' not detected. Running in generic mode (Serial Execution). Performance might be degraded.
> **警告**：未检测到 Sub-agent 'worker'。正在以通用模式（串行执行）运行。性能可能会受影响。
---
## 1. 标题
- **摘要**：2-4 行概述
- **要点**：
1. 要点一
2. 要点二
3. 要点三
- **来源**：[链接](URL)
- **关键词**：`keyword1` `keyword2` `keyword3`
- **评分**：⭐⭐⭐⭐⭐ (5/5)
---
## 2. 标题
...
---
*Generated by Daily News Report v3.0*
*Sources: HN, HuggingFace, OneUsefulThing, ...*
Constraints and Principles
- Quality over quantity: Low-quality content does not enter the daily report
- Early stopping mechanism: Stop scraping once there are enough 20 high-quality items
- Parallelism first: SubAgents in the same batch execute in parallel
- Failure tolerance: A single source failure does not affect the overall workflow
- Cache reuse: Avoid repeatedly scraping the same content
- Main Agent control: All decisions are made by the Main Agent
- Fallback Awareness: Detect sub-agent availability; gracefully degrade when unavailable
Expected Performance
| Scenario | Expected Time | Description |
|---|---|---|
| Best case | ~2 minutes | Tier1 is sufficient, no browser needed |
| Normal case | ~3-4 minutes | Tier2 supplementation needed |
| Browser needed | ~5-6 minutes | Includes JS-rendered pages |
Error Handling
| Error Type | Handling Method |
|---|---|
| SubAgent timeout | Log the error and continue to the next one |
| Source 403/404 | Mark as disabled, update sources.json |
| Content extraction failed | Return the raw content, Main Agent decides |
| Browser crash | Skip the source, log the event |
Compatibility & Fallback
To ensure usability across different Agent environments, the following checks must be performed:
-
Environment Check:
- During the Phase 1 initialization stage, try to detect whether the
workersub-agent exists. - If it does not exist (or the relevant plugin is not installed), automatically switch to Serial Execution Mode。
- During the Phase 1 initialization stage, try to detect whether the
-
Serial Execution Mode:
- Do not use parallel block.
- The main Agent executes the scraping task for each source sequentially.
- Although it is slower, it ensures that basic functionality remains available.
-
User prompt:
- A clear warning message must be included at the beginning of the generated daily report (in the blockquote section), alerting the user that it is currently running in degraded mode.