daily-press-scanner

Daily Press Scanner Skill

This skill is now a lightweight input layer and is no longer responsible for local article recognition and local summary generation.

Scope of responsibility:

1. The local script is only responsible for downloading PDFs, extracting the text layer, writing page-level text, and results.json
2. The subsequent AI is responsible for identifying key articles, ranking, summarizing, and expanding individual articles

The applicable target isChinese-language machine-translated newspaper PDFs. The goal is to give the AI in the chat a clean, stable, low-dependency input, rather than rebuilding the entire newspaper with local rules.

Main entry point

The only maintained entry point is:

python3 daily-press-scanner/scripts/extract.py \
  --source-config daily-press-scanner/configs/sources.example.json \
  --out-dir ./out \
  --run-date 2026-03-29

A single URL is also supported:

python3 daily-press-scanner/scripts/extract.py \
  --url "https://dl.dengtazk.xin/%E3%80%90%E8%AF%91%E3%80%91%E7%BA%BD%E7%BA%A6%E6%97%B6%E6%8A%A5-3-29.pdf" \
  --out-dir ./out

A URL list is also supported:

python3 daily-press-scanner/scripts/extract.py \
  --urls urls.txt \
  --out-dir ./out

scan.py Only a compatibility wrapper is retained, and it should no longer be used as the main entry point.

Input method

Recommended --source-config, suitable for daily automation.

Example:

{
  "sources": [
    {
      "source_name": "New York Times",
      "url_template": "https://dl.dengtazk.xin/%E3%80%90%E8%AF%91%E3%80%91%E7%BA%BD%E7%BA%A6%E6%97%B6%E6%8A%A5-{month}-{day}.pdf",
      "enabled": true
    }
  ]
}

Supported placeholders:

• {year}
• {month}
• {day}
• {date}

If --run-dateis not passed, the current date is used by default.

Prerequisites

Required:

• python3
• pdftotext Usually comes from poppler, and is the preferred text layer extractor.
• pymupdf When pdftotext is unavailable or fails, it automatically falls back.

Script behavior

The script only does these things:

1. Parse --url / --urls / --source-config
2. Download the PDF
3. Preferentially use pdftotext to extract the text layer
4. Automatically fall back to PyMuPDF
5. on failure; write page-level text to text/<paper_slug>/page-XXX.txt
6. Write out results.json
7. Delete the local working PDF after processing each newspaper

The script does not do:

• Local article segmentation
• Local key-article ranking
• Local summary generation
• Default OCR main pipeline

Output directory

out/
  results.json
  text/
    <paper_slug>/
      page-001.txt
      page-002.txt
      ...

results.json is the main contract, and at minimum includes:

• run_date
• inputs
• papers

Each paper at minimum includes:

• source_name
• paper_id
• url
• source_record
• status
• method
• pages
• page_list
• text_dir
• error

How to use in chat

Browse Mode

The first answer should preferentially return a "key article list", and should not first spread out an entire long article.

Each article should at minimum include:

• Number
• Title
• Page number
• 1-2 sentence introduction

If the article title is not entirely clear, it can be described as "topic + page number", but the selectable list structure should still be maintained.

Read Mode

After the user names a certain article:

1. Return the organized body text, long summary, or structured key points of that article
2. Attach the "brief article list" again at the end of the answer

This tail list should at minimum retain:

• Number
• Title
• Page number
• Very short introduction

The purpose is simple: after the user finishes reading one article, they do not need to scroll up for a long time to continue selecting the next article.

Article Retrieval

In browsing mode, the AI first uses text/<paper_slug>/page-*.txt to identify and rank key articles on its own.

When the user requests expanding a certain article:

1. Find the corresponding title and page number based on the number
2. Return to the relevant page text
3. Extract and organize the body text of that article
4. Return the organized body text result
5. Attach the brief article list again at the end

If a page contains multiple articles, prioritize locating within the page based on titles and context, rather than dumping the entire page's full text to the user as-is.

Context compression strategy

When the chat context approaches the limit, it must be compressed according to this priority:

1. Prioritize keeping the list of featured articles
2. Prioritize keeping a short introduction for each article
3. Prioritize keeping numbers, page numbers, and title mappings
4. Prioritize compressing the details of individual long articles that have already been displayed

In other words, it is better to compress an already expanded article into 5-8 bullet points than to lose the article list.

Recommended answer pace

1. First give the list of featured articles in the current issue of the newspaper
2. When the user clicks an article, expand that article
3. After expanding, repost the brief list at the end
4. The user continues by clicking the next article

This skill is suitable for a reading flow of "browse first, then dig deeper," and is not suitable for outputting the full text of the entire newspaper all at once.

Error handling

• The failure of a single newspaper does not affect the whole batch
• Download failures or extraction failures will be written to results.json
• Regardless of success or failure, the working PDF should be deleted after processing ends

Non-goals

• Alignment with the English original text
• Precise cross-page stitching
• OCR as the default main process
• Directly producing the final daily report summary in a local script