SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsdaily-papers-fetch

daily-papers-fetch

Writing & research

View repository

Updated 10-15 22:14

About this skill

Before starting: First say "Starting paper fetching 🐕" and state today's date. If in multi-day mode, state the fetching range.

Paper Fetching (Fetch + Score + Enrich)

You are the user's paper fetching system (step 1 of the 3-step pipeline). Fetch the latest papers → score and filter → enrich information → save to a temporary file.

Step 0: Read shared config

First read ../_shared/user-config.json, if ../_shared/user-config.local.json exists, use it to override the default values.

Explicitly generate and consistently use these variables afterward:

  • VAULT_PATH
  • DAILY_PAPERS_PATH
  • KEYWORDS
  • NEGATIVE_KEYWORDS
  • DOMAIN_BOOST_KEYWORDS
  • ARXIV_CATEGORIES
  • MIN_SCORE
  • TOP_N

Where:

  • DAILY_PAPERS_PATH = {VAULT_PATH}/{daily_papers_folder}
  • All keywords, categories, and thresholds are based on the shared config

Afterward, consistently use the shared config and the variables above as the basis.

Parse days

Parse from the user input the --days N parameter. Matching rules:

  • "last week", "last 7 days", "the week's papers" → --days 7
  • "last 3 days", "the last three days", "fetch 3 days" → --days 3
  • "last two weeks" → --days 14
  • No special specification / "run paper fetching" → do not add --days(default today)

Store the parsed number of days as the variable DAYS_ARG, and use it in subsequent script calls.

Config sources

  • The default config is at ../_shared/user-config.json
  • The personal override config is at ../_shared/user-config.local.json
  • If both exist, local takes precedence

Workflow

Phase 1+2: Fetch + Score + Merge and Deduplicate (pure Python script)

Use fetch_and_score.py to complete HF + arXiv fetching, scoring, merge deduplication, history deduplication, and selecting Top 30 in one step.Zero token consumption.

# 默认：当天
python3 ../daily-papers/fetch_and_score.py > /tmp/daily_papers_top30.json

# 多天模式（将 N 替换为解析出的天数）
python3 ../daily-papers/fetch_and_score.py --days N > /tmp/daily_papers_top30.json

Based on the previously parsed DAYS_ARG, if the user specified days, add --days N, otherwise do not add it.

The script automatically completes:

  • Parallel fetching of HuggingFace Daily + Trending API and arXiv API
  • Keyword scoring (positive/negative/domain bonus/trending bonus)
  • Merge and deduplicate by arXiv ID
  • Read .history.json for cross-day deduplication (including relaxed rules for weekend mode)
  • Backfill from history when fewer than 20 papers
  • Take Top 30 in descending order of score

Progress logs are output to stderr, JSON results are output to stdout.

Check output: Confirm that /tmp/daily_papers_top30.json exists and contains a valid JSON array. If it is an empty array or the file does not exist, check stderr to diagnose the problem.

Phase 3: Batch enrichment (enrich_papers.py script)

Use the enrich_papers.py script to enrich all papers at once. The script uses asyncio + curl subprocesses for concurrent requests, parses HTML with pure regex, no WebFetch needed.

First save the Top 30 results from Phase 2 to a temporary file, then run:

python3 ../daily-papers/enrich_papers.py /tmp/daily_papers_top30.json /tmp/daily_papers_enriched.json

Note: usetwo file path arguments (input + output), to avoid stdout/stderr confusion in the sandbox environment. The script treats the first .json argument as the input path and the second as the output path; if only one .json is passed, it will be treated as the input path, and the results go to stdout.

The script automatically does the following (Semaphore(10) limits concurrency, 30-second timeout per paper):

  • Fetch HTML pages + PDF pages in parallel
  • Extract from HTML: figure_url, authors, affiliations, section_headers, captions, has_real_world, method_names, method_summary
  • Extract from PDF: affiliations (via pdftotext | extract_affiliations.py）
  • If HTML authors is empty, fall back to the abs page <meta> tag to extract authors/affiliations
  • Merge priority (handled internally by the script):
    • figure_url: HTML curl
    • affiliations: PDF > HTML > abs fallback > Phase 1 data
    • authors: HTML > abs fallback > Phase 1 data
    • Other fields: HTML regex extraction

Output format: the same JSON array as the input, with the following fields added to each paper:

  • figure_url (string): first figure URL
  • affiliations (string): list of institutions, comma-separated
  • authors (string): list of authors (may be overwritten by a more complete source)
  • section_headers (array): section headers
  • captions (array): figure and table captions
  • has_real_world (bool): whether it includes real experiments
  • method_names (array): list of method names
  • method_summary (string): method description (300-500 characters)

Output

After completion, check that /tmp/daily_papers_enriched.json exists and contains a valid JSON array. Tell the user:

  • How many papers were fetched
  • How many were successfully enriched
  • Suggest running the next step:跑一下论文点评

Notes

  • Phase 1+2 use the fetch_and_score.py script,do not start a Task Agent, zero token consumption
  • Phase 3 uses the enrich_papers.py script, also does not start a Task Agent
  • If the script fails to execute, check the stderr output to diagnose the problem
  • If the arXiv API fetch fails, the script automatically falls back to the HuggingFace source only
  • If the total number of papers is fewer than 20, process as many as there are
  • Weekend strategy: arXiv does not update on weekends, and HF daily is basically empty on weekends, but HF trending updates continuously. On weekends, mainly rely on the trending source
  • Do not perform git operations, do not generate a recommendation file, only output temporary JSON