daily-news

Daily News

Three-stage workflow:Fetch metadata → Generate summaries → Output briefing

Working directory

On first run, ask for the working directory path (e.g. ~/daily-news), and remember it afterwards.

<workspace>/
├── profile.yaml          # 用户画像（关于我、关注什么）
├── settings.yaml         # 日报设置（语言、格式偏好）
├── methods/              # 信源获取方法
├── data/news.db          # SQLite 数据库
└── output/YYYY-MM-DD.md  # 日报输出

Initialization:

mkdir -p <workspace>/methods <workspace>/data <workspace>/output
cp references/examples/settings.example.yaml <workspace>/settings.yaml
cp references/examples/profile.example.yaml <workspace>/profile.yaml
python3 scripts/db.py init --db <workspace>/data/news.db

After initialization completes:

Write the working directory to the user's ~/.claude/CLAUDE.md, appending a line:
```
- daily-news skill 的项目目录在：<workspace>
```
This way, subsequent new sessions do not need to ask for the directory location again.
Ask the user whether the profile needs adjustment.

Stage 0: Select the fetch date range (new)

Before performing the fetch, ask the user for the date range, to reduce unnecessary work at the source.

Date range options

请选择抓取时间范围：
1. 今天（默认）                    → published_at >= 今天
2. 昨天                            → published_at >= 昨天
3. 最近3天                         → published_at >= 3天前
4. 最近7天                         → published_at >= 7天前
5. 从上次抓取至今（增量）          → published_at >= last_fetched_date
6. 自定义日期范围                  → 输入开始日期

Incremental fetch logic

Track the fetch date independently for each source：

# methods/twitter-karpathy.yaml
source_id: twitter-karpathy
...

# 自动维护的元数据
last_fetched_date: "2026-01-27"    # 上次抓取日期
last_fetched_count: 5               # 上次抓取数量
total_items_fetched: 127            # 累计总数

Database records：

source_status table: quickly query the last fetch date
source_sync_log table: detailed sync log

Date filtering execution flow

# 1. 获取上次抓取日期
python3 scripts/db.py source-status --db <db> --source <source_id>

# 2. 执行增量抓取
# 在 method 执行时传入 since 参数

# 3. 记录同步日志
python3 scripts/db.py add-items-incremental \
  --db <db> \
  --source <source_id> \
  --items '<json>' \
  --since "2026-01-27"

Stage 1: Fetch metadata (incremental optimization)

Iterate over the <workspace>/methods/ directory and execute each method file.

Incremental fetch check

For each method, before execution check:

# 1. 读取 method 文件中的 last_fetched_date
python3 -c "
import yaml
with open('methods/<source>.yaml') as f:
    config = yaml.safe_load(f)
print(config.get('last_fetched_date', 'never'))
"

# 2. 或查询数据库
python3 scripts/db.py source-status --db <db> --source <source_id>

Execute according to the date range

Case A: the method has a last date, and the user chose "incremental"

# 只抓取 last_fetched_date 之后的内容
# 在 method 执行时传入 since 参数

Case B: the method is being fetched for the first time, or the user chose "today/last N days"

# 使用用户指定的日期范围

Two-layer deduplication mechanism

First layer: filter during fetch(Stage 1)

# 在 method 执行时，只返回 published_at > since 的内容

Second layer: check before insertion(Stage 1.5)

# 批量检查 URL 是否已存在
python3 scripts/db.py check-existing \
  --db <db> \
  --urls '["url1", "url2", ...]'

# 返回已存在的 URL 列表

Third layer: database UNIQUE constraint(final protection)

-- items 表的 url 字段有 UNIQUE 约束
INSERT INTO items ...  -- 重复 URL 会触发 IntegrityError

Insert and record

python3 scripts/db.py add-items-incremental \
  --db <db> \
  --source <source_id> \
  --items '<json>' \
  --since "2026-01-27"

This command will:

Batch-check whether URLs exist (pre-deduplication)
Insert new entries
Record sync logs to the source_sync_log table
Update the source_status table's last_fetched_date
Update the metadata fields of the method file

Browser MCP check and automatic configuration

Before execution, first check whether any source needs Browser MCP (extends: browser-smart or detail_method: browser）：

If so, execute the following check flow:

1. Check whether MCP is already configured

# 检查 ~/.claude.json 中是否有 browsermcp 配置
grep -q '"browsermcp"' ~/.claude.json && echo "configured" || echo "not configured"

If not configured, add automatically:

# 读取现有配置，在 mcpServers 中添加 browsermcp
# 使用 Python 或 jq 修改 JSON
python3 -c "
import json
config_path = '$HOME/.claude.json'
with open(config_path, 'r') as f:
    config = json.load(f)
if 'mcpServers' not in config:
    config['mcpServers'] = {}
if 'browsermcp' not in config['mcpServers']:
    config['mcpServers']['browsermcp'] = {
        'command': 'npx',
        'args': ['@browsermcp/mcp@latest'],
        'type': 'stdio'
    }
    with open(config_path, 'w') as f:
        json.dump(config, f, indent=2)
    print('Browser MCP 配置已添加，请重启 Claude Code 生效')
else:
    print('Browser MCP 已配置')
"

2. Test connection status

调用 mcp__browsermcp__browser_snapshot 测试连接

Determine the result:

Returns containing Page URL: → connected, continue execution
Returns No connection to browser extension → not connected
Returns mcp__browsermcp__browser_snapshot is not a function or a similar error → MCP not loaded

3. Handle according to status

MCP not loaded (configuration just added):

Browser MCP 配置已添加。请：
1. 重启 Claude Code（Ctrl+C 退出后重新运行）
2. 重启后再次运行日报生成

MCP loaded but not connected:

Browser MCP 未连接到浏览器。请：
1. 安装 Chrome 扩展：https://chromewebstore.google.com/detail/bjfgambnhccakkhmkepdoekmckoijdlc
2. 在 Chrome 中点击扩展图标，点击 "Connect" 按钮
3. 然后重新运行

或者跳过这些信源，只处理 RSS/WebFetch 信源？

The user can choose to skip or wait for the connection and then continue.

Must use mcp__browsermcp__*, because it reuses the user's browser login state and can access content that requires login. Other browser tools (playwright/chrome-devtools) start an independent instance and cannot use already logged-in accounts.

Method execution logic

Read the metadata of the method file, and based on the extends field, determine the execution method:

Has extends (references a common method):

# extends: rss（最快，有 RSS 源时首选）
python3 references/methods/rss.py --url "<source_url>"

# extends: webfetch-smart（快，适用大多数网站）
# 读取 references/methods/webfetch-smart.md 按指引操作

# extends: browser-smart（Browser MCP，JS渲染/需登录）
# 读取 references/methods/browser-smart.md 按指引操作
# 注意：需要用户先点击 Browser MCP 扩展连接

No extends (fully customized):

*.py: directly execute python3 <file>
*.md: read the content, and operate the browser according to the instructions

Save to database

python3 scripts/db.py add-items \
  --db <workspace>/data/news.db \
  --source <source_id> \
  --items '<JSON>'

Stage 2: Generate summaries

python3 scripts/db.py list-pending --db <workspace>/data/news.db

For each piece of content, based on the detail_method field of the method file, obtain the body text:

detail_method	Retrieval method
`fetch`	WebFetch tool (fast)
`browser`	Browser MCP (slow, requires manual connection)
Not specified	Default `fetch`

Generate summaries according to references/prompts/summary.md , and update the database:

python3 scripts/db.py update-summary \
  --db <workspace>/data/news.db \
  --id <item_id> \
  --data '<摘要JSON>'

Stage 3: Generate daily report

python3 scripts/db.py list-today --db <workspace>/data/news.db

Read <workspace>/profile.yaml, and generate the daily report according to references/prompts/report.md . Output to <workspace>/output/YYYY-MM-DD.md。

User profile

The profile is used to evaluate content relevance. It is a natural language description with no fixed format.

Initial setup

A blank profile.yamlwas created during initialization. Ask the user whether they need to adjust it:

What do you do?
What have you been paying attention to recently?
Is there anything you don't want to see?

Update <workspace>/profile.yaml。

Update profile

When the user says "update profile":

Ask what has changed recently
Work direction? New interests?
Update profile.yaml

Profile format

about: |
  （关于我：身份、工作）

focus: |
  （当前关注：最近在意的话题）

low_priority: |
  （不太关心：降低优先级的内容）

Add source

Try in order of priority:

Check RSS
- Try common paths such as /feed、/rss、/atom.xml and other common paths
- or check the page <link rel="alternate" type="application/rss+xml">
- if present, use extends: rss
Test WebFetch
- Use the WebFetch tool to retrieve the page and see whether the article list can be extracted
- if successful, use extends: webfetch-smart
Use the browser
- WebFetch fails (JS rendering/anti-scraping/login required)
- Use extends: browser-smart
- Use Browser MCP to directly use the user's browser login state

Create a method file; see references/schemas/method.md

Reference materials

Material	Path	Load timing
Method specification	`references/schemas/method.md`	When adding a source
Summary prompt	`references/prompts/summary.md`	Stage 2
Daily report prompt	`references/prompts/report.md`	Stage 3
Common method	`references/methods/`	Stage 1 (referenced by extends)
Daily report settings example	`references/examples/settings.example.yaml`	Initialization
Profile format reference	`references/examples/profile.example.yaml`	During initialization
Website template	`references/website-template/`	When creating a website
Method metadata example	`references/examples/method-with-metadata.example.yaml`	Incremental scraping configuration reference

Database command reference

Incremental scraping related

# 批量检查 URL 是否已存在
python3 scripts/db.py check-existing --db <db> --urls '["url1", "url2"]'

# 增量添加条目（自动记录日志）
python3 scripts/db.py add-items-incremental \
  --db <db> --source <id> --items '<json>' --since "2026-01-27"

# 查看信源同步状态
python3 scripts/db.py source-status --db <db> --source <id>

# 查看同步日志
python3 scripts/db.py sync-log --db <db> --source <id> --limit 10

Database migration

# 升级到 V2（添加增量抓取支持）
sqlite3 <db> < scripts/migrate_v2.sql

Dependencies

pip install pyyaml feedparser requests beautifulsoup4

The browser method (browser-smart) uses Browser MCP and requires installing the Chrome extension.

Website deployment (optional)

After the daily report is generated, it can be automatically deployed to the website.

Quick start

Use the built-in website template:

# 复制模板到工作目录
cp -r references/website-template <workspace>/website
cd <workspace>/website
python3 build.py

Directory Structure

<workspace>/
├── output/              # 日报 Markdown 输出
└── website/             # 网站项目
    ├── build.py         # 构建脚本（来自模板）
    ├── dist/            # 生成的静态网站
    └── README.md        # 部署指南

Template Location

Template Path: references/website-template/
Included Files:
- build.py - Convert Markdown to terminal-style HTML
- README.md - Deployment Guide

Template Features

Terminal Style - Black header + white content area
Responsive Design - Adapts to mobile and desktop
Date Navigation - Supports switching between historical daily reports
Zero Dependencies - Pure Python, no third-party libraries required

Deployment Platform Support

Cloudflare Pages (Recommended)
GitHub Pages
Vercel
Netlify
Any static hosting service

Creating a Website for the First Time

Automatically executed when selecting "Create Website":

Copy template:

cp -r references/website-template <workspace>/website
cd <workspace>/website
python3 build.py

Initialize Git:

git init
git add -A
git commit -m "Initial commit"
gh repo create daily-news-web --public --source=. --push

Configure Cloudflare Pages:
- Build command: python3 build.py
- Build output: dist

Process After Daily Report Generation

After Stage 3 is complete, check whether the website directory exists:

Case A: website does not exist

Ask the user whether they need to create a website:

日报已生成：output/2026-01-28.md

是否需要部署到网站？
1. 是，创建网站（Cloudflare Pages）
2. 否，仅保存 Markdown

Case B: website already exists

Ask the user how to update:

日报已生成：output/2026-01-28.md

网站更新选项：
1. 立即构建并推送（自动部署到 Cloudflare）
2. 仅构建，稍后手动推送
3. 不更新网站

Automatic Push Command

Executed when selecting "Build and Push Now":

cd <workspace>/website
python3 build.py
git add -A
git commit -m "Add daily report for $(date +%Y-%m-%d)"
git push origin main

Cloudflare Pages detects the main branch push and automatically redeploys.

Manual Push Process

If the user chooses to push manually later:

# 随时手动执行
cd <workspace>/website
python3 build.py                    # 重新构建网站
git add -A
git commit -m "Add report for YYYY-MM-DD"
git push origin main                # 触发 Cloudflare 部署

Website Configuration

Configuration required when creating a website for the first time:

GitHub Repository: Push the website directory to GitHub
Cloudflare Pages：
- Connect GitHub repository
- Build command: python3 build.py
- Build output: dist
Custom Domain(optional): Add it in the Cloudflare Pages settings

For detailed configuration, see <workspace>/website/README.md

Custom Templates

To modify the website style, edit the CSS variables in <workspace>/website/build.py .

The original template is kept in references/website-template/ for reference.