data-sources

Data Sources

When to Use This Skill

Use when:

the user needs a public sports data source and is not sure which one;
choosing grain (game, team-game, player-game, play/event) before loading;
comparing nflverse / sportsdataverse / pybaseball / other public options;
writing a source plan with provenance, snapshot, and first-load checks.

Do not use this skill when:

the user already has a usable table and wants EDA or modeling;
they explicitly want the optional repository toolkit panels/CLI → sports-ds-bridge;
they already chose a loader and just need that loader skill.

Need	Go instead
NFL loader details	`nflreadpy`
Multi-sport SDV details	`sportsdataverse-py`
MLB / Statcast details	`pybaseball`
Optional toolkit bridge	`sports-ds-bridge`
EDA after load	`eda-sports`

Outcome

Produce a source plan naming the required grain, decision-time fields, primary source, fallback, coverage risks, terms, snapshot rule, and first-load checks. Source choice does not create predictive validity; it determines whether the question can be answered reproducibly and legally.

Use this skill before acquisition code or whenever coverage, field semantics, historical depth, IDs, latency, or licensing are uncertain. If the source is already chosen and a loader fails, use its package skill instead.

Required inputs

sport, competition, question, target, and analytical population;
analytical grain and natural key: season, game, team-game, player-game, possession, play, pitch, or event;
each candidate source's native grain and the aggregation/join contract needed to reach the analytical grain;
historical depth, expected volume, refresh cadence, and completion latency;
exact prediction decision time T and fields that must be known by then;
stable identifiers and cross-source joins required;
license, attribution, redistribution, authentication, and rate constraints;
required raw and derived artifact formats.

Candidate guide

These are ecosystems to investigate, not durable rankings. Confirm current coverage, access, and terms in upstream documentation at selection time.

Need	Candidate ecosystem	Verify before committing
NFL schedules, PBP, rosters, weekly data	nflverse / `nflreadpy`	table-specific grain, season coverage, schema drift
Basketball, hockey, college, soccer	maintained league/public modules, including SportsDataverse where applicable	competition/table support and field definitions
MLB Statcast/pitch or season tables	public MLB data tooling such as `pybaseball`	query-specific grain, bounded dates, field availability
Detailed event/possession data	public league-specific event provider	event IDs, ordering, correction policy
Cross-source enrichment	sources with stable IDs and timestamps	join coverage and as-of legality

Read the source matrix while shortlisting ecosystems, the terms notes before acquisition or redistribution, and the sanity checks before accepting the first pull.

Choose a source whose native records contain the facts required for the analytical grain, and write the aggregation contract when the grains differ. For example, play data may be aggregated to team-game only after event ordering, completion, and key rules are explicit. A season aggregate cannot manufacture play-level detail or historical publication vintages it never contained.

Source comparison rubric

Compare at least two plausible sources on:

Dimension	Questions
Coverage	Which competitions, seasons, phases, and entities exist?
Grain	What is one native record and its key? If it differs from the analytical grain, is the transformation explicit and valid?
Semantics	Are fields, units, ties, overtime, and completion states defined?
Time	Event, publication, update, and revision timestamps available?
Identity	Stable event/team/player IDs across seasons and sources?
Reliability	Bulk releases, versioning, uptime, schema-change notices?
Access	Terms, license, attribution, auth, rate and redistribution limits?
Reproducibility	Can responses be snapshotted and checksummed?
Cost/volume	Can the bounded request fit time, memory, storage, and quotas?

Do not treat “popular” or “easy to import” as sufficient evidence.

Workflow

Lock question, analytical grain/key, target, population, and T.
List only the minimum required raw fields and stable identifiers.
Shortlist and compare a primary and fallback source with the rubric.
Read upstream documentation and record field/time semantics and terms.
Estimate request count and data volume; bound event-level pulls by dates/IDs.
Pull the smallest representative sample spanning relevant eras/statuses.
Validate keys, row counts, periods, outcomes, entities, timestamps, and nulls.
Test required joins on a small sample and quantify unmatched/many-to-many rows.
Save the untouched raw response or immutable snapshot with retrieval metadata.
Document fallback triggers, known gaps, and the EDA handoff.

First-load diagnostics

nonzero rows and requested competitions/seasons are present;
natural keys are unique at the claimed grain;
scheduled, live, completed, canceled, and postponed records are distinguishable;
completed events have plausible outcomes and nonconstant score distributions;
team-game overall win rate is near its paired identity and home-only rates are plausible;
identifiers remain stable or have a documented crosswalk;
timestamps have known timezone and meaning;
critical missingness is measured by season/status/source;
source/version/query/retrieval time accompany every saved artifact.

key = ["game_id", "team"]
assert len(df) > 0
print(df["season"].value_counts().sort_index())
print("duplicate key rows:", df.duplicated(key).sum())
print(df.groupby("status")["game_id"].nunique())
print(df.isna().mean().sort_values(ascending=False).head(20))

Reject corrupt or mis-grained pulls rather than adapting the question silently.

Snapshot and provenance policy

Preserve raw responses before normalization. Record source/project version, endpoint or release, exact parameters, requested/retrieved times in UTC, schema fingerprint, row count, checksum, timezone, license/attribution, and any later correction policy. Derived tables should point to immutable raw snapshots and transformation code/config. Never overwrite data referenced by an experiment.

For predictive work, a current bulk snapshot may contain revisions unavailable historically. State whether the study reconstructs real-time vintages or accepts revised-history data and limit claims accordingly.

Hard constraints and anti-patterns

Do not	Why
scrape when a supported release/API exists	fragile and may violate terms
merge display names without a crosswalk	silent false matches and duplicates
use current roster/rating as historical	post-T information leaks backward
run unbounded pitch/play/event pulls	rate, memory, and reproducibility risk
switch sources mid-experiment silently	changes population and semantics
mix grains without an aggregation contract	estimand and independence change
infer season coverage from one scoreboard call	endpoint purpose is different
model before representative checks	corrupt data becomes plausible output

Respect terms, rate limits, access controls, and attribution. Never evade a provider restriction. Cache only when permitted.

Worked source plans

NFL pre-kickoff win model: team-game grain; nflverse schedules/results via nflreadpy; require stable game/team IDs, scheduled start, home/away, completion state, and scores for labels. Check schema across requested seasons and snapshot the release. Use a documented public fallback only if coverage fails.

MLB pitch analysis: pitch grain; pybaseball Statcast; bound every request by date and test a small window. Verify game_pk, at-bat/pitch order, units, missing tracking fields, doubleheaders, and revision behavior before scaling the pull.

Cross-source injury enrichment: select sources with stable player/team IDs and publication timestamps. Build a reviewed crosswalk, measure unmatched joins, and require an as-of predicate. If historical timestamps are absent, the source cannot support the intended pre-event claim.

Source-plan artifact

Question / sport / competition / target:
Analytical grain and natural key / population / decision time T:
Required fields and IDs:
Primary source and rationale:
Primary native grain/key and aggregation contract:
Fallback and trigger:
Fallback native grain/key and compatibility with primary:
Coverage window / status handling:
Event, publication, revision semantics:
License, terms, attribution, access limits:
Estimated request/volume and bounded query:
Representative sample checks:
Join checks and crosswalk:
Raw snapshot location / checksum / retrieved_at:
Known gaps and claim limitations:
EDA handoff:

Create the portable stub with:

python /path/to/data-sources/scripts/print_source_plan.py --out data/source_plan.md

Integrity and resource routing

Never claim coverage until a representative sample is inspected.
Preserve raw snapshots and provenance for every reproducible claim.
Source timestamps and revision policy are part of feature legality.
Read source_matrix.md for ecosystem choice, tos_notes.md for access and redistribution, and sanity_checks.md before accepting data.
Hand off loaded data to eda-sports, then feature-rules and validation.