Outcome
Create a versioned operating contract tied to user-owned model, data, metrics, feature, and validation artifacts. A model card describes an already evaluated model: what it is for, how it was tested, where it fails, and when to retrain or retire it.
A card is not a marketing page, experiment history, or substitute for evidence.
Use experiment-log for the sequence of trials and results-reporting for the
human-readable story of a particular evaluation.
When to use this skill
Use it when:
- a model has stable held-out evidence worth keeping;
- promoting an accepted experiment into a named version;
- preparing a model for reuse, handoff, review, or publication;
- updating the documentation for a new immutable model version;
- the user asks to document, freeze, or share a sports model.
Do not create a polished card to imply maturity that the model has not earned. If validation, a baseline, or decision-time legality is unknown, state that explicitly and mark the model unapproved for operational use.
Required evidence
- stable model name and version;
- owner, reviewers, creation date, and next review date;
- intended users, decisions, and prohibited uses;
- sport, competition, population, grain, target, and decision time;
- data provenance, window, filters, exclusions, and immutable snapshot;
- feature definitions and availability timing;
- named naive and strong simple baselines;
- validation design, fold boundaries, and metric definitions;
- aggregate and fold-level results on identical held-out populations;
- calibration, leakage, fairness, stability, and slice findings as applicable;
- serialized model, environment information, and reproduction instructions;
- monitoring, retraining, and retirement criteria.
Use unknown, not tested, or not applicable when truthful. Never fill a gap
with an inferred value or a generic promise.
Required sections
- Identity, ownership, status, and version
- Intended use, users, decisions, and prohibited uses
- Target, grain, prediction timestamp, horizon, and output semantics
- Data sources, window, population, exclusions, and snapshot
- Feature families, transformations, and time-safety rules
- Baselines and candidate family
- Validation design, fold construction, and metric definitions
- Results with uncertainty, fold variation, slices, and calibration
- Known limitations, failure modes, and misuse risks
- Monitoring, retraining, review, and retirement rules
- Artifact manifest, dependencies, and reproduction instructions
- Approval history and linked experiments
For a fill-in structure, read
references/card_template.md. Before defining
monitoring or retirement rules, read
references/kill_conditions.md and adapt the
examples to the actual decision and metric.
Workflow
- Confirm that the candidate met the predeclared acceptance rule on held-out data. If not, document it as experimental or abandoned rather than approved.
- Verify the model version and immutable artifact identifiers.
- Copy factual fields from experiment, data, feature, leakage, calibration, and validation artifacts; do not reconstruct them from memory.
- State intended use narrowly and list explicit prohibited uses.
- Describe data coverage, exclusions, label construction, and populations not represented by evaluation.
- Document each feature family and prove availability at decision time.
- Present candidate and baselines on identical folds and rows.
- Summarize the primary metric first, then uncertainty, calibration, stability, and error slices.
- Convert known failure modes into checkable monitoring and kill conditions.
- Record exact artifact paths and reproduction instructions supplied by the user or environment.
- Have a second reviewer trace every quantitative and operational claim.
- Freeze the version; future substantive changes create a new card.
Status and freeze rules
| Event | Card action |
|---|---|
| First candidate meeting the charter | create version v1 and mark evaluated |
| Feature definition or target changes | new major or minor version; new card |
| Data window or source snapshot changes | new version and fresh validation |
| Hyperparameter change | new version linked to the originating experiment |
| Editorial correction only | revise in place with dated correction note |
| Kill condition fires | mark retired or abandoned; preserve prior evidence |
| Reproduction no longer works | mark non-reproducible until repaired and reviewed |
Freeze only when data snapshot, feature set, configuration, serialized model, and validation results are immutable. Never silently edit a frozen card to make history look better.
Result presentation
| Claim | Required evidence in the card |
|---|---|
| Better win probabilities | log-loss or Brier versus baseline on same folds |
| Useful probability levels | calibration curve/table and calibration error context |
| Better margin prediction | MAE or RMSE versus constant/simple rating baseline |
| Stable team ranking | out-of-time rank utility and rank stability |
| General across seasons | per-season results, not aggregate mean alone |
| General across populations | evaluated slice results and coverage statement |
State metric direction. “0.681 versus 0.693” is incomplete unless the card says lower log-loss is better, gives sample size, and identifies the held-out period.
Monitoring and kill conditions
Conditions must be observable, bounded, and connected to a response. Good forms include:
- primary metric loses to the locked baseline for two consecutive review windows;
- calibration error exceeds a specified threshold for a minimum sample size;
- a required field's missingness or schema changes invalidate a feature family;
- a leakage audit becomes unresolved or fails;
- a rule, schedule, roster, or measurement regime changes beyond evaluated scope;
- population coverage moves outside the documented range;
- reproduction fails from the pinned artifacts and supported environment.
For each trigger, record owner, check cadence, minimum evidence, grace period, and action: investigate, retrain, restrict, roll back, or retire. Avoid conditions such as “when performance feels bad.”
Sports-specific caveats
- State whether the model scores games, team-games, player-games, possessions, plays, or pitches. A doubled team-game panel is not game-level independence.
- Define treatment of ties, overtime, postseason events, neutral venues, and canceled or incomplete contests.
- Document schedule-strength, expansion, rule-era, roster, and source changes that may limit transport across seasons.
- Pre-event models must state the exact cutoff and how late injury, lineup, starter, weather, or market information is handled.
- Team or player identifiers must be stable across relocations and name changes.
- If probabilities drive decisions, document calibration and any threshold or utility assumptions; accuracy alone is insufficient.
Hard constraints and integrity rules
- Never describe training metrics as expected performance.
- Never omit the baseline, decision time, grain, or primary metric.
- Never claim generality beyond evaluated sports, seasons, and populations.
- Never publish a card without artifact locations, an owner, and review date.
- Never leave retirement criteria implicit.
- Never hide leakage, validation, or calibration status.
- Never describe exploration as production-ready.
- Never list a reproduction command that does not match the represented version.
- Never overwrite a frozen model history with a better-looking later result.
- If required evidence is unavailable, state the operational consequence.
Anti-patterns
- “Works well” with no metric, baseline, period, or denominator;
- a card written before any ordered held-out evaluation exists;
- one aggregate score with losing folds omitted;
- NFL wording, assumptions, or commands copied into another sport's card;
- kill conditions that cannot be computed;
- a feature list without as-of availability;
- a generic “bias reviewed” statement with no slice, definition, or result;
- a model filename that cannot be connected to its data and configuration.
Worked examples
Pre-game team-win probability
Identity: home_form_logit_v3
Status: evaluated; not approved for wagering
Purpose: Estimate pre-event P(team wins) from venue and shifted form
Grain: team-game; paired rows share one contest
Decision time: scheduled start, before event-day updates
Data: completed regular-season games, 2018-2024; immutable Parquet snapshot
Validation: season walk-forward; primary metric log-loss
Baseline: constant training-fold prevalence
Result: report mean and every held-out season on identical rows
Leakage: checked; all rolling features shifted before aggregation
Limit: no injury or lineup model; cold start in early season
Kill: retire after two eligible seasons worse than baseline log-loss
Artifacts: data snapshot, feature manifest, fold metrics, predictions, model file
Margin model with incomplete calibration scope
Identity: pregame_margin_ridge_v1
Purpose: Predict home-team final margin before lineup lock
Primary metric: MAE versus historical-mean and simple-rating baselines
Calibration: not applicable to point predictions; interval coverage not tested
Status: experimental until prediction-interval coverage is evaluated
The second example does not manufacture a calibration claim; it states the missing uncertainty check and constrains status accordingly.
Helper
python <path-to-model-card>/scripts/write_card_stub.py \
--name home-form-logit \
--version v1 \
--grain team-game \
--out data/model_card.md
Fill the generated user-owned Markdown file from verified artifacts. The helper creates structure only; it does not validate performance or approve the model.
Output contract
A complete card has:
- every required section present;
- baseline and primary metric beside candidate results;
- aggregate, fold-level, and applicable slice evidence;
- decision time and time-safety status;
- explicit leakage and calibration status;
- concrete monitoring, retraining, and kill conditions;
- linked experiments and immutable artifact identifiers;
- exact reproduction instructions;
- version status, owner, reviewers, and next review date.
Resources
references/card_template.md— read when drafting the full card section by section.references/kill_conditions.md— read when converting failure modes into measurable monitoring and retirement rules.scripts/write_card_stub.py— portable model-card scaffold writer.