SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsdata-pipeline-manager

data-pipeline-manager

Data & analytics

View repository

Updated 10-13 14:49

About this skill

Data Pipeline Manager

Make deliberate, documented data decisions before training. This skill prevents the most common experimental validity failures: undocumented preprocessing, leaky splits, contaminated test sets, and irreproducible data pipelines.

Use this skill when:

  • deciding how to acquire, access, or download a dataset
  • designing or auditing train/val/test split protocols
  • preprocessing choices need documentation before they affect results
  • data quality issues could corrupt baselines or ablations
  • contamination between training and evaluation data is a reviewer risk
  • dataset versions, seeds, and pipeline steps need to be pinned for reproducibility
  • a paper claim depends on a specific data protocol and the protocol is not yet written down

Do not use this skill for running experiments — use run-experiment once the pipeline is validated. Do not use this skill to write result sections — use experiment-story-writer after results exist.

Pair this skill with:

  • experiment-design-planner to ensure the split protocol matches the experimental design's evaluation criteria
  • baseline-selection-audit when dataset choice or protocol affects which baselines are comparable
  • research-project-memory to record data decisions as claims, evidence, and risks
  • result-diagnosis when surprising results may trace to a data artifact rather than a method failure
  • paper-evidence-board when data protocol details need to appear in the paper's methods and evidence slots
  • init-python-project when the project's data/ layout and pipeline scripts need to be scaffolded

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
├── references/
│   ├── split-protocols.md
│   └── contamination-guide.md
└── templates/
    └── pipeline-plan.md

Progressive Loading

  • Always read references/split-protocols.md when designing or auditing split protocols.
  • Read references/contamination-guide.md when handling LLM evaluation, pre-trained model probing, retrieval benchmarks, or any setting where the model may have seen test-adjacent data.
  • Read templates/pipeline-plan.md before writing a pipeline plan.
  • Read memory/claim-board.md and memory/evidence-board.md when data decisions need to be linked to paper claims.
  • Read code/.agent/benchmark-plan.md or equivalent when a benchmark protocol already exists.

Core Principles

  • Treat data decisions as experimental controls, not implementation details.
  • Document every preprocessing step before it runs, not after results appear.
  • Design splits from the evaluation goal backward: what question must the test set answer?
  • Never let the model, hyperparameter search, or any paper-motivated decision touch the test set.
  • Contamination is the paper's responsibility: assume reviewers will ask, and have an answer.
  • Version the processed dataset alongside the code that created it.
  • Seed-fix every shuffle, sampling step, and augmentation that affects reproducibility.
  • Data quality issues found after training are ten times more expensive than those found before.

Step 1 — Gather Context

Locate:

  • project root: look for data/, datasets/, code/data/, or user-provided path
  • existing pipeline scripts: data_utils.py, preprocess.py, scripts/prepare_data.sh, or equivalent
  • existing documentation: docs/data/, data/README.md, or any dataset description
  • experiment design: code/.agent/benchmark-plan.md, memory/claim-board.md
  • paper methods draft: paper/sections/experiments.tex or equivalent when present

Record:

  • what dataset(s) are in use or planned
  • whether a split already exists
  • what preprocessing is already applied or planned
  • whether any data quality audit has been done
  • whether contamination has been checked

If nothing exists, produce a draft pipeline plan and mark uncertain fields as TBD.

Step 2 — Select Pipeline Mode

Choose based on the user's request or the current gap:

  • audit: review an existing pipeline for completeness, reproducibility, and reviewer risk
  • split: design or validate the train/val/test split protocol
  • quality: run or plan a data quality audit (missing values, label noise, distribution shifts, class imbalance)
  • contamination: check for data leakage between splits and pre-training / retrieved data
  • versioning: pin dataset version, hashes, download scripts, and processing seeds
  • plan: create a full pipeline plan covering all of the above for a new dataset

Default: use audit when a pipeline exists, plan when none exists.

Step 3 — Design or Audit the Split Protocol

Read references/split-protocols.md.

For every dataset split, record:

- Split: train | val | test | held-out
- Size: N examples (% of total)
- Stratification: class-stratified | time-ordered | random | grouped | domain-separated
- Overlap policy: none | explicit-allowed (state why)
- Leakage controls: [list controls]
- Assignment seed: <integer>
- Purpose: what hypothesis or metric this split answers
- Forbidden use: what must NOT touch this split before final evaluation

Flag any split designed after the method was partially tuned — this is a reviewer-visible risk. Flag any split where the validation set was used as a proxy test set.

Step 4 — Data Quality Audit

Check:

  • Missing values: are they meaningful (masked) or errors? What is the imputation or exclusion policy?
  • Label noise: known label error rate, labeling protocol, inter-annotator agreement if applicable
  • Class imbalance: imbalance ratio, whether baselines handle it equivalently
  • Distribution shift: between train and test, or between this dataset and cited prior work
  • Duplicate or near-duplicate examples: deduplication policy and its effect on split boundaries
  • Outliers: what is excluded and on what criteria — is this reproducible?
  • Data leakage from preprocessing: normalization fit on train only; no augmentation at test time

For each finding, classify as:

  • blocker: will cause invalid results or comparison failures
  • risk: should be disclosed in the paper's limitations or experiment settings
  • note: worth tracking but not blocking

Step 5 — Contamination Check

Read references/contamination-guide.md.

For LLM evaluation, retrieval, or pre-trained model probing:

  • Check whether benchmark test examples appear in common pretraining corpora (e.g., via n-gram overlap, exact hash, or known contamination audits)
  • Check whether model selection used test performance directly or indirectly
  • Check whether retrieved context at test time could contain test-set answers
  • Check whether data augmentation generated test-adjacent examples

For standard supervised tasks:

  • Confirm no sample appears in both train and test after deduplication
  • Confirm that grouping or stratification respects natural data boundaries (e.g., speaker IDs, document IDs, patient IDs)
  • Confirm that any held-out reserve set was never used for threshold selection or model comparison

For every contamination finding, record the severity and the available mitigation.

Step 6 — Version and Reproducibility Record

For reproducible experiments, the pipeline plan must include:

- Dataset name and version: <name>==<version> or commit/hash
- Download source: <url or repo> (verified checksum or hash)
- Raw data hash: sha256=<hash>
- Processing script: <path/to/script> at git commit <sha>
- Processing seed: <integer> for every random step
- Processed data hash: sha256=<hash> of final splits
- Storage location: <path or remote>
- Excluded samples: <rule and count>
- Known issues: <list>

If any of these fields cannot be filled, mark as TODO and add to memory/action-board.md.

Step 7 — Write the Pipeline Plan

Use templates/pipeline-plan.md.

Save to:

code/.agent/data-pipeline-plan.md

If no code/ directory exists, save to:

.agent/data-pipeline-plan.md

When updating, preserve stable decisions and add a compact change note.

Step 8 — Memory Writeback

After completing a pipeline plan or audit:

  • Write data decisions that affect claims to memory/claim-board.md
  • Write contamination risks to memory/risk-board.md
  • Write unresolved data quality issues as actions to memory/action-board.md
  • Write evidence about dataset characteristics to memory/evidence-board.md when they are cited in the paper
  • Update memory/current-status.md when the dataset version or split protocol changes

Step 9 — Route Follow-Ups

  • experiment-design-planner: align experimental design with the finalized split protocol
  • baseline-selection-audit: recheck baseline comparability after split or preprocessing changes
  • result-diagnosis: diagnose surprising results that may trace to a data issue
  • paper-evidence-board: add dataset protocol details to the paper's evidence slots
  • init-python-project: scaffold data/ layout and pipeline scripts

Final Sanity Check

Before finalizing:

  • every split has a documented purpose and a forbidden-use policy
  • every preprocessing step is reproducible with a fixed seed and a version-pinned script
  • data quality findings are classified as blockers, risks, or notes
  • contamination checks are explicitly done or explicitly deferred with a risk entry
  • dataset version, hashes, and download source are recorded or marked TODO
  • memory writeback covers affected claims, risks, and actions
  • next action is clear