SuperAPI AI Directory
Partner中文AI newsFavorites
Skillslight-data-engineering

light-data-engineering

Data & analytics

View repository

Updated 10-16 09:58

About this skill

Data Processing, Quality Analysis, and Dataset Construction

Core Principles

Data First: Before proposing an idea in m03, first answer the four questions—is the data sufficient to support the research? Is the quality reliable? Is the scale sufficient? Are the features worth mining? Avoid fantasy divorced from the data foundation.

Input / Trigger

  • Upstream (before idea, main line): User's raw data / ready-made dataset / self-built requirement → output the four-question conclusion and feed it to m03/m04.
  • Back edge (experiment stage, from m05): Can receive the derived data requirements from the "Derived Data Specification" block of the research-plan experiment matrix (base dataset + transformation type + key parameters + split strategy), and based on this produce the correspondingnoise addition/missing/cross-domain/parameter sweepevaluation set and dataset_card, backfill db04 for use by ROB/GEN/SEN experiments. Derived set construction follows the methods and anti-leakage iron rules in "Processing Flow", "Splitting", and "Self-built Dataset Planning" below.

Data Health Check (do first)

  1. Overview: rows and columns, types, memory, samples.
    • Small-to-medium data with pandas: upon reading, immediately set dtype/parse_dates，df.info(memory_usage='deep') to see real memory,df.isna().mean() to get the missing rate in one line,df.describe(include='all'). Convert object columns to category to save memory.
    • For large data, choose the engine by scenario: for single-machine speed + query optimization use a Polars lazy pipeline pl.scan_csv(...).filter(...).group_by(...).collect(), for extremely large open collect(streaming=True); to query parquet/csv directly with SQL and go out-of-core use DuckDB (duckdb.sql("SELECT ... FROM 'data/*.parquet'"), automatically spills beyond memory); to keep the pandas style while going beyond memory use Dask dd.read_csv("*.csv")...compute(). (The engine selection matrix takes a09 tool-selection decision_matrix.md as the single source of truth; the previously recommended Vaex has been unmaintained since 2023, do not newly adopt it.)
  2. Quality profiling: use each tool for its strengths as needed, don't run just one.
    • ydata-profiling produces an overall EDA report:ProfileReport(df, title=...).to_file("r.html"); when there are many columns/rows be sure to minimal=True, for time series tsmode=True; to compare before/after cleaning or train-test use r1.compare(r2). Look at the report Alerts (high correlation, high cardinality, constant, missing).
    • Deepchecks runs structured validation suites: afterDataset(df, label=, cat_features=[...]) then data_integrity().run(ds)(duplicates/mixed types/feature-label leakage/outliers),train_test_validation().run(train_ds, test_ds)(drift/new categories/train-test sample overlap leakage).
    • Great Expectations provides reproducible quality gates:gx.get_context() → Data Source/Asset/Batch → Expectation Suite（ExpectColumnValuesToNotBeNull/ToBeBetween/ToBeInSet/ToBeUnique/ToMatchRegex) → Validation Definition → Checkpoint, producing Data Docs. Note that GX 1.x and 0.x APIs are discontinuous, check the version.
    • For standardized exchange/publishing metadata use Frictionless:frictionless describe infers the Table Schema,frictionless validate datapackage.json produces a cell/row-level error report.
  3. Quality conclusion: give a clear judgment and rationale of "directly usable / needs cleaning / insufficient to support / needs supplementary collection".

EDA and Statistical Analysis Flow

  • EDA seven steps: structural overview → missing analysis (missingno matrix/heatmap to see missing patterns) → univariate (numeric histograms + boxplots, categorical frequencies) → bivariate (scatter + correlation, grouped boxplots, cross-tabs/chi-square) → multivariate (correlation heatmap, pairplot, PCA) → target relationship (features vs target, preliminary judgment of predictive power and leakage) → quality summary. EDA conclusions are only hypotheses; don't use the same data to both explore and draw statistical conclusions.
  • Statistical test decision: first check assumptions—normality (Shapiro-Wilk; for large samples switch to QQ plot, as it is overly sensitive), homogeneity of variance (Levene). Two groups numeric normal → t-test (Welch if variances unequal); non-normal → Mann-Whitney U; paired → paired t/Wilcoxon; multiple groups → ANOVA/Kruskal-Wallis; categorical association → chi-square/Fisher. Always report effect sizes (Cohen's d, Cramér's V, r/ρ) and confidence intervals, not just p. Multiple testing must be corrected: FDR (Benjamini-Hochberg) preferred, Bonferroni more conservative. Beware of p-hacking and HARKing.

Processing pipeline

  • Cleaning: deduplication, type correction, unified encoding/units, text normalization.
  • Missing values: distinguish mechanisms (MCAR/MAR/MNAR) and choose a strategy—deletion/mean-median/model imputation/indicator flag.
  • Outliers: IQR/z-score/isolation forest/business rules, distinguish errors vs true extremes.
  • Feature engineering: encoding, scaling, construction, selection (filter/wrapper/embedded), dimensionality reduction.
  • Data augmentation: choose tools by modality,split first then augment, augment only the training folds(validation/test keep the original distribution, otherwise metrics are inflated)—images albumentations (synchronized bbox/mask transforms), text nlpaug (synonyms/context/back-translation, be careful not to flip label semantics), time series tsaug (TimeWarp/Drift/noise addition, be careful not to break causality and introduce leakage), tabular SMOTE etc. fit only on training folds. For modality-specific usage and red lines see the references "Data Augmentation" section.
  • Splitting (scikit-learn):train_test_split(..., stratify=y, random_state=) classification must be stratified; choose cross-validation by data nature—for time series use TimeSeriesSplit(prevent future leakage), for repeated individuals use GroupKFold/StratifiedGroupKFold(prevent group leakage), for general classification use StratifiedKFold. Iron rule against leakage: scaling/imputation/encoding/feature selection all go inside Pipeline+ColumnTransformer, fit only on training folds, never fit_transform on the full data before splitting. Record random seeds.

Planning a self-built dataset

Collection method, annotation guidelines, data format, data documentation (datasheet), privacy compliance (de-identification/authorization/licensing), release method (Zenodo/HF/Figshare + DOI + license). Produce a dataset_card (see db04 fields).

  • Use template for annotation guidelines templates/annotation_guide.md: category definitions / boundary case decision rules / LLM-assisted annotation + human review loop / QC sampling rate / IAA consistency metrics. Consistency metricsreuse code_assets/agreement.py(Cohen's κ / quadratic weighted κ for ordinal ratings / Fleiss' κ for ≥3 annotators / ICC(2,1) for continuous values, already aligned with sklearn), do not rewrite; if κ<0.6 it is not up to standard and you must go back to supplement boundary rules and re-annotate.
  • LLM pre-annotation cannot be directly treated as ground truth: it amplifies model bias, must have a human review loop, keep an audit trail pred_label/human_label/是否修正/仲裁。

Find existing datasets (check before building your own)

  • OpenML: lightweight data retrieval fetch_openml(name=, version=, as_frame=True) or fetch_openml(data_id=); full capabilities openml.datasets.get_dataset(id).get_data(target=)、list_datasets(...), standardized tasks/runs reproducible. Lock version/data_id to ensure reproducibility.
  • Hugging Face Datasets：load_dataset(name, split=), local load_dataset("csv"/"parquet", data_files=), very large streaming=True(IterableDataset). For processing use .map(batched=True)/.filter/.train_test_split。
  • Kaggle：kaggle.json place ~/.kaggle/(permissions 600),kaggle datasets list -s 关键词、kaggle datasets download -d owner/name --unzip. You must first accept the dataset terms; note each dataset's license.

Quality assessment metrics

Completeness, consistency, accuracy, timeliness, uniqueness, representativeness, bias (bias_risk), privacy (privacy_risk), annotation quality. For drift monitoring use the new Evidently API:Dataset.from_pandas(df, data_definition=DataDefinition(...)) wraps current/reference,Report(metrics=[DataDriftPreset()]).run(current_data=, reference_data=)，.save_html(). Drift tests are selected per column (KS/PSI/chi-square/Wasserstein); note that Evidently's API has changed drastically in recent years, so check the version, and "drift detected ≠ harmful" requires business judgment.

  • Annotation quality needs not just metrics but detection methods: use cleanlab confident learning to locate suspected mislabeled samples—cross-validation out-of-sample predicted probabilities → find_label_issues produces a list sorted by suspiciousness (usage see references "cleanlab" section). pred_probs must be out-of-sample; after finding candidatesmanually adjudicate top-K, do not fully auto-delete. cleanlab (locating specific suspicious samples) and IAA (assessing the overall reliability of the annotation process,code_assets/agreement.py) are complementary; do both.

Outputs

  1. Data quality report (including the four-question conclusions).Standard artifacts:quality_report.md(naming see CONVENTIONS §6.1).
  2. Reproducible processing pipeline (scripts + parameters + seeds, handed to a03 for implementation).
  3. Split scheme and explanation.
  4. dataset_card (when self-built).Project-level data card standard artifact:data_card.md(hand to m05/a03/m06).

Interfaces

Conclusions feed into m03/m04; pipeline handed to a03 for implementation; dataset registration db04 and project repository db09. Privacy/license issues escalate to a10.

Bundled skill scripts (directly runnable, pure python + synthetic self-tests, no network dependency)

All scripts include --selftest(no data needed, built-in synthetic data + assertions verifying the detector actually triggers).

  • scripts/data_doctor.py: CSV → Markdown data health check report (shape/types/true memory/missing/duplicates/constant columns/all-empty columns/high cardinality/IQR outliers/strong correlation/target leakage hints, with issues summarized by HIGH/MED/LOW).
    • Self-test:python scripts/data_doctor.py --selftest
    • Usage:python scripts/data_doctor.py --csv data.csv --target y --out report.md（--sample N sample first to prevent large tables from freezing).
  • scripts/safe_split.py: by --task clf/reg/timeseries/group build Pipeline+ColumnTransformer(numeric median imputation + standardization, categorical most_frequent imputation + OneHot) and automatically select CV—StratifiedKFold/KFold/TimeSeriesSplit/GroupKFold/StratifiedGroupKFold. Built-in leakage assertions: prove preprocessing is refit separately in each fold (in-fold mean ≠ full-data mean), eliminating fit_transform before splitting.
    • Self-test:python scripts/safe_split.py --selftest(all four task types run).
    • Usage:python scripts/safe_split.py --csv data.csv --target y --task group --group-col user_id。
  • scripts/quality_gate.py: validate CSV against YAML rules, output PASS/FAIL report (dataset: min/max_rows, no_duplicate_rows; columns: dtype/required/non_null/unique/min/max/enum/regex). Pure pandas+PyYAML, no hard dependency on GX/Frictionless, exit code 0/1 can be used directly as a CI data gate.
    • Self-test:python scripts/quality_gate.py --selftest; rule examples see examples/rules.example.yaml。
    • Usage:python scripts/quality_gate.py --csv data.csv --rules rules.yaml --out gate.md。
  • assets/data_card_template.md: datasheet template (fields aligned with db04), including quality assessment, bias/privacy risks, access tiering, recommended splits, provenance verification section.
  • scripts/check_access_level.py: data access tiering gatekeeper. Each dataset/derived set declares in its data card access_level（raw/redacted/verified_only), the script verifies whether it can flow to a downstream sink—raw data flowing to public stages such as paper/figure/public-repo will beblocked(nonzero exit code, usable as a pipeline gate). Tristate pass/blocked/unknown, judged only by declarations; whether actual de-identification is adequate still requires a10 review.
    • Self-test:python scripts/check_access_level.py --selftest
    • Usage:python scripts/check_access_level.py --level raw --sink paper, or --manifest flows.json to batch-validate the flow destination list.
  • For statistical tests/effect sizes/multiple correction, reuse the repository root's code_assets/stats_tests.py(relative to this skill it is ../../code_assets/, including welch_t, benjamini_hochberg, wilson_ci, etc.); for annotation consistency reuse agreement.py; for long-tail resampling reuse longtail_resample.py; do not reinvent them.

For the tools' real endpoints/APIs/parameters and known pitfalls, see references.md(per-tool verification notes).