SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsauthoritative-data-harvester

authoritative-data-harvester

Data & analytics

View repository

Updated 10-16 04:57

About this skill

Authoritative Data Auto-Acquisition (Authoritative Data Harvester)

Global Workflow Collaboration Constraints (long-conversation drift prevention)

  • This skill must not be used as an isolated entry point. When the user requests a complete paper, Word generation, continuation of the workflow, or the stage is uncertain, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
  • Before starting or continuing a formal task of this skill, you must run: 
    python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill authoritative-data-harvester
  • If the output [WORKFLOW FAIL] or the report status != "PASS", stop this skill, and backfill the preceding stages according to paper_output/qa/workflow_guard_report.json 's failed items; do not continue from memory.
  • This skill only writes paper_output/ artifacts within its own contract scope; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and the next step.
  • In long conversations, if the context grows long, the stage is uncertain, or the user invokes skills separately, first run: 
    python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
    Then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and this skill's upstream JSON contract, and continue according to the report's recommended_skill and next_action .
  • Before continuing the workflow, you must treat paper_output/context/workflow_memory.json as a long-term breakpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report takes precedence.
  • After completing this skill's artifacts each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory: 
    python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
    After updating, read paper_output/context/workflow_memory.json / .mdto confirm the next step and recommended skill have been recorded.

Execution contract

  • Upstream input: the variable requirements given by the user, or paper_output/step1/problem_analysis.json、paper_output/plan/model_route.json 's identified external data requirements.
  • Required output: a reproducible data source description, scraping or download plan, and save the raw/processed data and source information to crawled_data/, preferably including crawled_data/sources.json。
  • Downstream handoff:data-cleaning-and-visualization reads crawled_data/ to perform unified cleaning, chart planning, and paper-grade figures.
  • Recommended next step: after data is persisted, proceed to data-cleaning-and-visualization; for a complete paper goal, return to paper-workflow-orchestrator to determine subsequent stages.
  • Failure fallback: if automatic acquisition is impossible, provide same-tier authoritative alternative sources, scope differences, and manual download paths; do not use unsourced or non-citable data to pass off as authoritative data.

Objective

In mathematical modeling tasks, quickly find "authoritative, citable, reproducible" public data sources, and obtain data in a way that minimizes web scraping and prioritizes APIs/bulk downloads, ultimately outputting:

  • Data acquisition script/plan (including links, parameters, time range, field explanations)
  • Raw data and cleaned data (CSV/Parquet)
  • Data dictionary and citation information (source, update time, license/terms, access date)

When to invoke

  • When authoritative/official data sources are needed (statistics bureaus, international organizations, government open data)
  • When a reproducible data scraping workflow is needed (fixed API parameters, repeatable execution)
  • When a variable list already exists but data is missing, or alternative indicators need to be supplemented

General principles (must be followed)

  1. Prioritize official APIs or Bulk Download; use HTML parsing only as a last resort.
  2. Comply with robots.txt and the terms of service, respect rate limits; do not bypass login/payment/captcha.
  3. The entire process must be reproducible: record the source URL, API parameters, access date, version/update time, field meanings, and units.
  4. Data quality first: align definitions, units, frequency, and geographic scope; clearly define strategies for handling missing and anomalous data.

Standard workflow (output according to this every time it is executed)

1) Requirement normalization

Output a "data requirements table" that includes at least:

  • Variable names (Chinese and English), units, expected frequency (daily/monthly/yearly), time range
  • Geographic granularity (country/province/city/grid) and definition notes
  • Allowed substitute indicators (when the primary indicator is unavailable)

2) Source selection strategy (authoritative first, convenience second)

Prioritize matching by task type:

  • Macro/development: World Bank, IMF, OECD, UNData
  • Population/society: UN DESA, World Bank, national statistical offices/censuses
  • Public health: WHO (when necessary, supplement with secondary aggregation sources and annotate the source chain)
  • Weather/climate: NOAA, NASA (choose according to open policies)
  • European Union statistics: Eurostat
  • China: National Bureau of Statistics, ministry open data, local statistical bureaus (prefer downloadable/API options)

3) Access method decision tree

  • Official API available: use the API directly
  • No API but bulk download available (CSV/Excel/ZIP): direct-link download
  • Webpage tables only: first look for the XHR/JSON behind the page; if that still does not work, then do HTML parsing

4) Scraping implementation specifications

The delivered script must have:

  • Parameterization: start/end, region, indicator, output_dir
  • Robustness: retries (exponential backoff), timeouts, rate limiting, caching/resume from checkpoints
  • Persistence: separate raw/ and processed/ directories; retain original responses or original files
  • Logging: record only necessary information, do not output sensitive information

5) Cleaning and validation (minimum requirements)

  • Fields: unified naming (snake_case), type conversion, unit conversion, time index alignment
  • Missing data: explain the reason for missingness (unavailable/gap/definition change), and provide a handling strategy
  • Anomalies: basic rule validation (range, threshold for year-over-year/month-over-month jumps)
  • Spot checks: compare sample rows against source pages/metadata

6) Deliverables checklist (must be output)

  • Data files: raw.*, processed.csv (or parquet)
  • Metadata: sources.json (name/url/access_method/params/license/updated_at/accessed_at)
  • Data dictionary: field meanings, units, frequency, geographic granularity, missing-data strategy
  • Citation format: reference entries that can be used directly in papers/reports

Commonly used authoritative data sources (extensible)

  • World Bank Data API
  • IMF Data
  • UNData / UN agencies
  • OECD Data
  • Eurostat
  • WHO
  • NOAA / NASA
  • National statistical offices and government open data platforms

User input template (for quick start)

  • "I want to do [topic] modeling, with variables [A,B,C]; time [YYYY-YYYY]; region [country/province/city]; please provide authoritative sources and a reproducible data acquisition script + cleaning results."
  • "I need official data for [a certain indicator], preferably via API; if not, bulk download; please output sources.json + processed.csv."

Failure fallback strategy

When a source is unavailable/restricted:

  1. Switch to an authoritative source at the same level (e.g., UN ↔ World Bank ↔ OECD)
  2. Switch to an alternative indicator and clearly specify the differences in definitions
  3. Deliver only the most authoritative downloadable version, and explain why automated retrieval is not possible

Directory conventions (aligned with the project globally)

  • Raw downloads and API responses are recommended to be saved to:crawled_data/raw/。
  • Cleaned structured data is recommended to be saved to:crawled_data/processed/。
  • Source and reproducibility information is recommended to be saved as:crawled_data/sources.json(including url, parameters, access date, license/terms).

Before and after linkage

  • Subsequent steps usually begin with:data-cleaning-and-visualization(unify and clean the data in crawled_data/ and produce charts).
  • To continue to a paper draft: return to paper-workflow-orchestrator。

Constraints (must be followed)

  • Memory Interaction (required):
    • After obtaining data: must call context-memory-keeper, record the "new data source name" and "storage path" to Short-term Workbench。
  • This skill is only responsible for "obtaining authoritative data and ensuring reproducibility"; it is not responsible for directly writing the paper body; outputs must be written to disk in crawled_data/。
  • If the data is to enter the paper, two points must be satisfied at the same time:
    • crawled_data/sources.json record citable source information in.
    • Subsequently call data-cleaning-and-visualization, clean the data and generate paper_output/figures/ evidence charts.