authoritative-data-harvester

Authoritative Data Auto-Acquisition (Authoritative Data Harvester)

Global process collaboration constraints (long-conversation drift prevention)

• This skill must not be used as an isolated entry point. When the user requests a complete paper, generates Word, continues the process, or the stage is uncertain, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
• Before starting or continuing the formal tasks of this skill, you must run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill authoritative-data-harvester
  

• If outputting [WORKFLOW FAIL] or reporting status != "PASS", stop this skill, and backfill the preceding stages according to the failure items in paper_output/qa/workflow_guard_report.json ; do not continue from memory.
• This skill only writes the paper_output/ artifacts within its own contract scope; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and the next step.
• In a long conversation, if the context becomes long, the stage is uncertain, or the user invokes skills separately, first run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
  

  then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and this skill's upstream JSON contract, and continue according to the recommended_skill and next_action in the report.
• Before continuing the process, you must treat paper_output/context/workflow_memory.json as a long-term breakpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report shall prevail.
• After completing this skill's artifacts each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory:

  python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
  

  After updating, read paper_output/context/workflow_memory.json / .mdto confirm that the next step and recommended skill have been recorded.

Execution contract

• Upstream input: the variable requirements given by the user, or paper_output/step1/problem_analysis.json、paper_output/plan/model_route.json the external data requirements identified in.
• Required output: a reproducible data source description, scraping or download plan, and save the raw/processed data and source information to crawled_data/, preferably including crawled_data/sources.json。
• Downstream handoff:data-cleaning-and-visualization read crawled_data/ for unified cleaning, chart planning, and publication-quality figures.
• Recommended next step: after the data is persisted, proceed to data-cleaning-and-visualization; for complete paper goals, return to paper-workflow-orchestrator to determine subsequent stages.
• Failure fallback: if automatic acquisition is not possible, provide same-tier authoritative alternative sources, differences in definitions, and manual download paths; do not use unsourced or non-citable data to pass off as authoritative data.

Goal

In mathematical modeling tasks, quickly find "authoritative, citable, reproducible" public data sources, and obtain data in a way that avoids web scraping as much as possible and prioritizes APIs/bulk downloads, ultimately outputting:

• Data acquisition script/plan (including links, parameters, time range, field explanations)
• Raw data and cleaned data (CSV/Parquet)
• Data dictionary and citation information (source, update time, license/terms, access date)

When to invoke

• When authoritative/official data sources are needed (statistics bureaus, international organizations, government open data)
• When a reproducible data scraping process is needed (fixed API parameters, repeatable execution)
• When a variable list already exists but data is missing, or alternative indicators need to be supplemented

General principles (must be followed)

1. Prefer official APIs or bulk downloads, and only use HTML parsing as a last resort.
2. Comply with robots.txt and terms of service, respect rate limits; do not bypass login/payment/captcha.
3. The whole process must be reproducible: record source URL, API parameters, access date, version/update time, field meanings and units.
4. Data quality first: align definitions, units, frequency, geographic scope; clarify missing and anomaly handling strategies.

Standard workflow (output according to this every time it is executed)

1) Requirement normalization

Output a "data requirements table", containing at least:

• Variable name (Chinese and English), unit, expected frequency (daily/monthly/yearly), time range
• Regional granularity (country/province/city/grid) and definition notes
• Allowed substitute indicators (when the primary indicator is unavailable)

2) Source selection strategy (authority first, convenience second)

Prioritize matching by task type:

• Macro/development: World Bank, IMF, OECD, UNData
• Population/society: UN DESA, World Bank, national statistical offices/censuses
• Public health: WHO (when necessary, supplement with secondary aggregated sources and annotate the source chain)
• Meteorology/climate: NOAA, NASA (choose according to open policies)
• EU statistics: Eurostat
• China: National Bureau of Statistics, ministry open data, local statistical bureaus (prefer downloadable/API)

3) Access method decision tree

• Official API available: direct API
• No API but bulk download available (CSV/Excel/ZIP): direct link download
• Webpage table only: first look for the XHR/JSON behind the page; if still not possible, then do HTML parsing

4) Scraping implementation specification

The delivered script must have:

• Parameterization: start/end, region, indicator, output_dir
• Robustness: retries (exponential backoff), timeout, rate limiting, caching/checkpointing
• Persistence: separate raw/ and processed/ directories; retain the original response or original file
• Logging: record only necessary information, do not output sensitive information

5) Cleaning and validation (minimum requirements)

• Fields: unified naming (snake_case), type conversion, unit conversion, time index alignment
• Missing values: explain the reason for missingness (unavailable/gap/definition change), and provide a handling strategy
• Anomalies: basic rule validation (range, year-over-year/month-over-month jump thresholds)
• Spot checks: compare sample rows with source pages/metadata

6) Deliverables list (must output)

• Data files: raw.*, processed.csv (or parquet)
• Metadata: sources.json (name/url/access_method/params/license/updated_at/accessed_at)
• Data dictionary: field meanings, units, frequency, regional granularity, missing-value strategy
• Citation format: reference entries that can be used directly in papers/reports

Common authoritative data sources (extensible)

• World Bank Data API
• IMF Data
• UNData / UN agencies
• OECD Data
• Eurostat
• WHO
• NOAA / NASA
• National statistical offices and government open data platforms

User input template (for quick start)

• "I want to do [topic] modeling, with variables [A,B,C]; time [YYYY-YYYY]; region [country/province/city]; please provide authoritative sources and reproducible data acquisition scripts + cleaning results."
• "I need official data for [some indicator], prefer API, otherwise bulk download; please output sources.json + processed.csv."

Failure fallback strategy

When a source is unavailable/restricted:

1. Switch to an authoritative source at the same level (e.g., UN ↔ World Bank ↔ OECD)
2. Switch to an alternative indicator and clearly specify the differences in definitions
3. Deliver only the most authoritative downloadable version, and explain why automated retrieval is not possible

Directory conventions (aligned with the project's global conventions)

• Raw downloads and API responses are recommended to be saved to:crawled_data/raw/。
• Cleaned structured data is recommended to be saved to:crawled_data/processed/。
• Source and reproducibility information is recommended to be saved as:crawled_data/sources.json(including URL, parameters, access date, license/terms).

Connection with before and after

• Subsequent work usually starts with:data-cleaning-and-visualization(unifying and cleaning the data in crawled_data/ and producing charts).
• If you want to continue to a paper draft: return to paper-workflow-orchestrator。

Constraints (must be followed)

• Memory Interaction (required):
  • After obtaining the data: you must call context-memory-keeper, recording the “new data source name” and “storage path” in Short-term Workbench。
• This skill is only responsible for “obtaining authoritative data and ensuring reproducibility,” not for directly writing the main text of the paper; the outputs must be persisted to disk in crawled_data/。
• If the data is to be used in the paper, two conditions must be met at the same time:
  • crawled_data/sources.json record citable source information in .
  • Subsequently call data-cleaning-and-visualization, clean the data and generate paper_output/figures/ evidence charts.