sports-modeling-doctrine

Sports Modeling Doctrine

Outcome

Write a modeling charter before acquiring data or fitting models. The charter is a user-owned source of truth for the question, target, timing, evaluation, and shape of done.

Before code, lock the scientific contract:

what question is being answered;
what one row represents;
what is knowable at decision time;
what baseline counts as evidence of value;
how success will be measured out of time;
what result causes acceptance, revision, or stopping.

If these fields are not named, do not fit a model.

When to use this skill

Use it when starting or reframing a sports analysis, deciding what “good” means, reviewing whether a result is evaluable, or preventing exploratory work from quietly becoming a predictive claim.

After the charter, use only the relevant specialist skills: a public-source loader for acquisition, eda-sports for data understanding, feature-rules for time-safe predictors, baseline-models for reference models, and the relevant statistical, predictive, rating, simulation, or validation skill.

Question types

Type	Core question	Required evidence	Common overreach
Descriptive	What happened?	coverage, denominators, uncertainty	treating the sample as every era/population
Explanatory	What is associated?	design, effects, uncertainty, confounding limits	causal language
Predictive	What will happen after time `T`?	time-safe inputs and ordered holdout	explaining coefficients as causes
Causal	What changes under an intervention?	identification strategy and assumptions	using prediction alone
Ranking	Who is strongest as of `T`?	future utility, schedule context, rank stability	treating point ranks as certain
Simulation	What distribution follows assumptions?	calibrated inputs and sensitivity	reporting simulated precision as observed fact

Do not mix explanatory and predictive success. A stable association need not improve forecasts, and a useful predictor need not identify a causal mechanism.

Charter workflow

Write the question in one sentence.
Name sport, competition, population, era, and exclusions.
Define grain: game, team-game, player-game, possession, play, pitch, or other.
Classify the analysis type using the table above.
For predictive, ranking, or prospective simulation work, define decision time T and forecast horizon precisely.
Define target or estimand, units, label rules, and treatment of ties or missing outcomes.
Record base rate, null expectation, or incumbent process.
Name at least one naive and one strong simple baseline.
Lock the primary metric and its direction before fitting.
Choose validation that respects event, season, and availability order.
Define data requirements, minimum coverage, and provenance standards.
Write acceptance, failure, revision, and stop conditions.
Record out-of-scope decisions and unsupported uses.

Grain and decision-time contract

For each proposed field, be able to answer:

row grain: <what one row represents>
entity/event key: <natural key>
decision time T: <timestamp or event boundary>
forecast horizon: <what future interval is predicted>
feature available by T: <yes/no and source timestamp>
target observed after T: <yes/no>
paired/dependent rows: <shared games, players, series, seasons>

Sports data routinely contain doubled team-game rows, postgame summaries, revised injury reports, end-of-season aggregates, and ratings updated after the event. Column names do not prove availability. Define an as-of rule.

Baseline ladder

Level	Example	Purpose
Null	constant training prevalence or historical mean	proves value beyond no differentiation
Structural	venue indicator, prior rank, or league average by context	captures obvious domain structure
Strong simple	shifted form, simple rating, or regularized linear/logistic model	tests whether complexity earns its cost
Incumbent	current operational forecast or public reference	measures practical replacement value

Lock baseline definitions, training windows, and missing-data behavior before candidate tuning. All models must be scored on identical held-out rows.

Metric defaults

Target	Primary candidates	Baseline	Important secondary evidence
Binary outcome	log-loss, Brier	prevalence, venue/simple rating	calibration and fold spread
Margin/continuous	MAE, RMSE	historical mean or simple rating	residual distribution and interval coverage
Count	Poisson deviance, MAE	historical mean count	dispersion and zero behavior
Time to event	proper survival score/concordance	simple survival estimate	calibration by horizon
Ranking	future-result correlation or utility	prior rank/rating	rank stability and uncertainty
Simulation	distributional score or coverage	simple empirical distribution	sensitivity to assumptions

Accuracy is rarely sufficient for probability work. Match the metric to the decision, define its direction, and compute it only on genuinely held-out data.

Validation defaults

Ordered sports events usually require expanding-window or rolling-origin validation. A default season walk-forward design trains on seasons before s and evaluates on season s. When a sport lacks clean seasons, split by event time with explicit gaps or embargoes where labels/features overlap.

Random row shuffles are not the default because they can mix future roster, team-strength, schedule, and feature information into training. Group shared contests together, and account for repeated teams or players when estimating uncertainty.

Acceptance and stopping rules

A predictive candidate is normally worth keeping only if it:

beats locked baselines on the mean primary metric;
improves in a meaningful majority of eligible folds or has a justified pooled result;
passes time-safety and leakage checks;
avoids catastrophic calibration if probabilities are quoted;
remains useful under relevant era, team, player, or event slices;
justifies its complexity and maintenance burden.

For explanatory work, require stable effect direction, honest uncertainty, and bounded claims. For simulation, require input calibration and sensitivity.

Read references/good_enough.md when setting the acceptance rule. Stop when the rule is met, when a failure condition invalidates the work, or when further complexity does not earn material held-out value.

Charter schema

question
sport_and_competition
population_and_exclusions
grain_and_natural_key
analysis_type
decision_time_and_horizon
target_or_estimand
base_rate_or_null
naive_and_strong_baselines
primary_metric_and_direction
secondary_metrics
validation_design
data_requirements_and_provenance
uncertainty_plan
acceptance_rule
failure_and_stop_conditions
out_of_scope_and_prohibited_uses
required_artifacts

Worked charters

Pre-game team-win probability

Question: Do venue and shifted form improve P(team wins)?
Sport/population: NFL completed regular-season games, 2018-2024
Grain: team-game; two paired rows per contest
Decision time: scheduled kickoff
Target: won; ties reported separately
Baselines: constant training prevalence; venue-only logistic
Primary metric: held-out log-loss, lower is better
Validation: season walk-forward; contest rows remain together
Acceptance: beat both baselines in mean and at least five of seven folds
Failure: any feature not provably available before kickoff
Out of scope: causal claims and wagering profitability

Player next-game count

Question: Predict a starter's strikeouts in the next start.
Grain: player-game
Decision time: first pitch
Target: strikeouts in that start
Baselines: trailing shifted mean; opponent strikeout-allowed mean
Primary metric: MAE
Validation: date walk-forward with player histories truncated at T
Cold start: separate rule and reporting slice

Read references/charter_examples.md for additional compact NFL, NBA, and MLB patterns.

Non-negotiables and integrity rules

Time order matters; random event shuffles need explicit justification.
Features must be legal at T for prospective claims.
Baselines are defined before candidate models.
The primary metric is locked before inspecting test folds.
Grain changes require explicit aggregation and key validation.
Candidate and baseline use identical held-out populations.
Failures and non-improvement are recorded as valid results.
Strong claims require a leakage audit and honest uncertainty.
Public data does not eliminate provenance, licensing, or snapshot duties.
Scope does not expand silently after favorable results appear.
Use the simplest model that meets the charter.
Record decisions and artifacts outside ephemeral chat.

Anti-patterns

choosing an algorithm before defining the decision;
feature soup without an availability timestamp;
calling one hot season “generalization”;
changing the primary metric after seeing held-out results;
treating each team-game row as an independent game;
copying the same assumptions across sports with different schedules or rules;
continuing to tune after the test window becomes familiar;
declaring victory because a complex model beats the null but not a strong simple baseline;
producing probabilities without a calibration plan.

Helper

python <path-to-sports-modeling-doctrine>/scripts/print_charter_template.py \
  --out data/modeling_charter.md

Fill the user-owned template before data acquisition or fitting. The helper does not choose the question, baseline, or acceptance rule.

Operating and handoff rules

Use only the skills relevant to the charter; every skill must stand alone.
Pass explicit user-owned artifacts between acquisition, analysis, and reporting steps.
Keep the charter current when a justified scope decision changes.
Re-charter rather than quietly changing target, grain, or decision time.
Report a blocker when timing, grain, provenance, or evaluation cannot be established.

For optional downstream routing, read references/skill_path.md. It is a decision aid, not a requirement to invoke every listed skill.

Output contract

Return or save the completed charter with question, type, sport/population, grain/key, decision time, target, baselines, primary metric, validation, uncertainty, acceptance/failure rules, out-of-scope uses, and required artifacts. Do not return an algorithm recommendation without this contract.

Resources

references/charter_examples.md — read for sport- and target-specific charter examples.
references/good_enough.md — read when defining acceptance, failure, and stopping rules.
references/skill_path.md — read when choosing optional specialist handoffs after the charter.
scripts/print_charter_template.py — portable charter writer.