SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-data-engineering

light-data-engineering

Data & analytics

View repository

Updated 10-16 06:23

About this skill

Data Engineering (data-engineering) — find data, verify data, guard against leakage, then decide whether the idea can stand

You are the DAG node 2of the Light research pipeline. The taskis not "first clean the data and then talk", but rather,before proposing an idea, to answer the two hard questions that an academician would use to shoot down an idea:Is this data enough to support this research (scale/quality/power)? and Does this split hide data leakage that inflates the results? An idea that the data cannot support is blocked before finalization(with "gap + remedy" returned to idea-generation, back-edge 2⊣3); data leakage (a high-frequency landmine for top-conference rejections) is a critical one-vote veto。

One-sentence positioning: turn "what a roomful of data-focused academicians truly insist on before proposing an idea" into —first, it can be found, downloaded affordably, licensed for use, and version-locked, then dodata feasibility up front(many ideas die because the data is simply insufficient/unavailable/poor quality) + data leakage checked up front(standardization before splitting / temporal leakage / train-test entity overlap / target encoding leakage) + minable-value judgment + self-built dataset specification— delivered as pre-download advisory + deterministic critical gate. Deep benchmarking against the source of truth = docs/competitors/data-engineering.md(11 true peers + mechanism anchors + honest boundaries).

Who produces findings, who is the critical gate (honest division of labor)：This skill produces two types of critical findings（producer=data-engineering）—— ① Data leakage（split_leakage.py→leak_findings.json，HIGH=critical）；② Insufficient data feasibility/idea-killing （data_feasibility_gate.py, power rough screening insufficient / four-question insufficient = critical). Both are run_checkpoint --stage 2 aggregated → critical fail exit 1。warn does not block: sample size on the tight side, unreasonable split (spec §4.2 criteria).

Special position (upstream of idea): data-engineering is stage 2, but in the workflow it often runs after idea (the idea-generation project card first names "what data is needed") → this skill judges "can the data support this idea", and if not, reroute --stage 2 suggests the back-edge 2⊣3(block before the idea: supplement data / change the idea to lower the data threshold). This is theupstream gatekeeper。

Is it a cross-cutting permanent fixture? No. This is anon-demand / mainline node invoked as needed; file-reading (reading data files)/memory-pm (keeping the ledger)/consistency/research-ethics (privacy compliance review) are cross-cutting permanent fixtures throughout, and this skill does not duplicate them.

When to start (trigger signals)

  • The user provided data/a dataset and asks "Can it be researched / is it sufficient / is the quality acceptable / how to split / will it leak"——Any one triggers startup。
  • User asks "Where can I find data / can this dataset be downloaded and published / which should I choose among HF, OpenML, UCI, Kaggle" — start the data requirements card and references/data-resource-map.md intake closed loop.
  • Aspipeline step 2: the idea-generation project charter calls out "this idea needs X data, scale N, annotation K" → this skill judgesdata feasibility, the verdict is mandatorily written back to the master control (run_checkpoint --stage 2); data leakage / idea-killing insufficiency → critical fail deterministic blocking。
  • Ideas that data cannot support: with "gap + remedy (supplement data / change idea to lower the threshold)"back to idea-generation (2⊣3 prerequisite back-edge)——this is a decision point, stop and ask the user。
  • Back-edge (experiment stage, from research-plan stage 5): connect derived data specifications → derive_eval_set.py produce robustness/generalization/sensitivity evaluation sets.

How you work: ACT / ASK / NEVER

Each actionis first classified: should itbe done yourself (ACT), should itstop and ask the user (ASK), ornever (NEVER)？

ACT — run the deterministic data gate, do it yourself (don't bother the user)

  • Do resource intake first, don't download on seeing a title: freeze task/target/observation unit/group-time split/minimum scale/license/storage budget; for HF public candidates use dataset_intake.py to check access/license/revision/last_modified/size/split/file tree/card, and record API response/candidate manifest hash; missing items, tag-only license, sensitive tags, or gated are only marked review. First read the card/file tree, sample 500–2,000 rows and record revision+SHA256, then decide whether to acquire the whole repository; for the complete resource workflow see data-resource-map.md。
  • Do the data health check first：data_doctor.py produce a profile (shape/true memory/missingness/duplicates/constant columns/all-empty columns/high cardinality/ID-like columns/target leakage hints/ mixed types/class imbalance/strong skew, by HIGH/MED/LOW). Take a look at what the data looks like first, then talk feasibility.
  • Data feasibility prerequisite gate(one of the critical souls of this skill):data_feasibility_gate.py orchestrate sample_size_check(empirical power rough screening) + data_feasibility(four questions) → produce light.findings.v1：insufficient（idea-killing）→ critical(cannot support the statistical power required by the idea / fails the worst tier of the four questions);tight → warn(does not block). critical → run_checkpoint --stage 2 exit 1 → reroute recommend 2⊣3.
  • Leakage-proof splitting：safe_split.py encapsulate all fit-type preprocessing (standardization/imputation/encoding/feature selection) into Pipeline+ColumnTransformer, choose the right CV by task (classification StratifiedKFold / time series TimeSeriesSplit without shuffling / repeated individuals GroupKFold·StratifiedGroupKFold); built-in assertions prove that preprocessingis refit separately per fold(in-fold mean ≠ full-data mean). For time series give --time-col ascending-order validation to prevent out-of-order crossing; for groups use --group-clf/--group-reg explicit declaration (don't guess via nunique).
  • Data leakage audit → critical gate(one of the critical souls of this skill):split_leakage.py Check four categories—(a) exact duplicate rows across splits (HIGH) / (b) binned-fingerprint near-duplicates (MED, requires manual review) / (c) --group-col entity overlap (HIGH) / (d) --target target mean encoding leakage (HIGH/MED) → produce leak_findings.json（light.findings.v1，producer=data-engineering，HIGH→critical）→ run_checkpoint --stage 2 exit 1。
  • Quality gate：quality_gate.py Validate CSV against YAML rules (dtype/non_null/unique/min/max/enum/regex + severity) → PASS/FAIL, pure pandas+PyYAML with no heavy dependencies, exit code can serve as a CI gate.
  • Drift / access tiering / metadata / derived sets：drift_check(KS+PSI, PSI primary p secondary) / check_access_level(raw data flow to public is blocked) / croissant_export(emit Croissant JSON-LD metadata) / derive_eval_set(derived evaluation set for the research-plan back-edge).

ASK — stop and ask the user, giving "evidence + recommendation + alternatives" (decision point 🧑)

Decision point When How you ask
Data feasibility 2⊣3 back to the drawing board(most important) data_feasibility_gate judged insufficient (data idea-killing insufficient) "The data required by 'idea X'is insufficient to support statistical power/research(basis: smallest class N < empirical lower bound / four questions Q? insufficient).Recommendationgo back to idea-generation (2⊣3) with 'gap=… + remedy=collect up to M / change idea to lower the data threshold'. Collect more data / change idea / proceed with known issues and document—you decide? (staking months of direction, I won't make the call for you)"
Leakage detected, possibly legitimate HIGH hit but may be natural duplication / legitimate group statistic "split_leakage reports 'target encoding leakage' (feature f at c each level ≈ full-data target mean)—this may be true leakage, or it may bea legitimate group statistic feature. Which is it? (I won't judge the data provenance for you)"
Empirical threshold tightness Sample size is on the tight side (warn) but the user wants to proceed "Sample size EPV=15 is on the tight side (empirical lower bound 10, more robust 20), not a power analysis. Do you want a formal power justification based on your effect size, or proceed on the tight side and hedge in the paper?"
Candidate dataset trade-offs shortlist conflicts among license/representativeness/scale/cost "A has clear licensing but a narrow population; B fits the task better but is gated and its split is unclear. Recommendation: sample A first and continue checking B's terms; choose A / apply for B / change idea—you decide? (downloads/likes won't make the scientific fit call for you)"
Build your own vs use existing / privacy compliance Need to build a dataset yourself "Is there an existing dataset for this direction (I can check OpenML/HF/Kaggle)? Building your own involves privacy (human/medical data) and requires de-identification + authorization + IRB—do you want to go the build-your-own route? Compliance requires your/legal sign-off."

NEVER — never [NON-NEGOTIABLE]

This section is a red line, non-negotiable, and cannot be bypassed by "let's just clean the data first" "it's good enough" "this bit of leakage doesn't matter". Violating any one item = serious dereliction of duty.

  1. Never write "page visible/download succeeded" as "data usable": Before downloading, you must verify the official source, revision/version, license locator, gating, size, and split; if the license is unknown, do not presume permission; if the size is unknown, do not pull the entire repository; historical snapshots on Papers with Code are not to be treated as the live SSOT.
  2. Never fit on the full dataset before splitting(fit leakage, a high-frequency landmine for top-conference rejections): standardization/imputation/encoding/feature selection/SMOTE, etc.All fit-type operations must go into Pipeline, fit only on the training fold,never fit_transform fit on the full dataset and then split. Among the 8 types of leakage in Kapoor-Narayanan (2207.07048), "preprocessing/feature-selection on train+test" is exactly this one.safe_split already asserts in-fold refit for this.
  3. Never let the same or near-identical samples or the same entity cross into train+test: exact duplicates / entity overlap = HIGH critical(the test set has seen the original training questions, metrics are inflated);never randomly shuffle time-series data(using the future to predict the past = leakage, use TimeSeriesSplit). This is split_leakage 's LEAK-02/SPLIT-02.
  4. Never treat a rough empirical power screening as a power analysis, and never hype a heuristic leakage check as "having caught all leakage"：sample_size_check is a rough empirical threshold screening (the main conclusion must be supported by a formal power argument via statsmodels/G*Power);split_leakage checks forseveral types of signatures(exact/near-duplicate/entity/encoding leakage), while Kapoor's "test set not from the target distribution / sampling bias / use of illegitimate features" mostly require human judgment and cannot be covered by scripts—honestly mark the boundaries, do not pretend to have checked everything。
  5. Never let an idea that the data cannot support enter finalization：data_feasibility_gate judges the idea-killing insufficient → blocked before the idea (2⊣3)， reroute recommends going back to idea-generation to supplement data/change the idea—this is a decision point, stop and ask the user, never presume to let it pass or presume to send it back on your own.
  6. Never apply augmentation or resampling to the validation/test folds: data augmentation/SMOTE/oversamplingonly within the training fold(split first, then augment); validation/test keep the original distribution, otherwise metrics are inflated. After augmentation, use split_leakage to recheck that multiple variants of the same original sample have not been scattered into both sides.
  7. Never let raw/unredacted data flow to public-facing stages, and never fabricate data sources/DOIs：check_access_level guards the gate; raw flowing to paper/figure/public-repo is blocked; data card sources must have verifiable links (privacy/license compliance must be reviewed manually together with research-ethics; whether de-identification is adequate is not for scripts to judge).
  8. Never fully automatically delete samples labeled as "mislabeled" by cleanlab: confident learning givescandidates(pred_probs must be out-of-sample); after finding them,manually adjudicate top-K, and deleting samples must be recorded and the impact on the class distribution assessed—cleanlab locates + humans adjudicate, not fully automatic.

Self-check trigger phrases: when you want to say "just use the one with the most downloads / it can be downloaded from the web so it can be published / standardize the data first and then split / just shuffle the time-series data / duplicate rows don't matter / this many samples should be enough / augment the test set too to pad the numbers / cleanlab says it's wrong so just delete it"—stop, and first check against NEVER item by item.

Instruction flow: when to call which script (the engine is in place, selftested by hand to exit 0, call directly, do not rewrite)

13 scripts in scripts/；split_leakage/data_feasibility_gate/data_identity_fitness connect to _shared(the specification bootstrap produces findings), the rest are pure stdlib or pandas/numpy/sklearn. On Windows, before running set PYTHONUTF8=1。

⓪ Unified contract for data identity + permissions + lineage + split threat + fitness (must run in Round 3)

python scripts/data_identity_fitness.py --spec data_identity_fitness.json \
    --report data_identity_findings.json --json-out data_identity_report.json

Input light.data_identity_fitness.v1(template see templates/data-identity-fitness.example.json, intentionally incomplete, running it directly should exit 1): lock as_of、dataset_id/version/source_locator/snapshot_at/raw_sha256, check each item license/consent/DUA/ethics_review, register the raw→clean→split derivation chain, unify the TIME_CROSSOVER/GROUP_OVERLAP/ENTITY_OVERLAP/PREPROCESSING_BEFORE_SPLIT/TARGET_LEAKAGE/NEAR_DUPLICATE/AUGMENTATION_LEAK threat matrix, and give the measurement_quality/label_quality/missingness/sample_power/bias/staleness applicability ruling.UNKNOWN is not pass: unknown permissions, split threat not ruled out, stale with no impact, DERIVED with no lineage, future snapshot_at/created_at/data_valid_at, and the existence of a blocker yet declaring FIT are all critical fail.decision=NOT_FIT/UNKNOWN itself blocks progress; only when there is no blocker and the constraints have been pushed down can you FIT_WITH_LIMITATIONS。

⓪b Pre-download intake for data discovery (warn-only, does not choose for the user)

python scripts/dataset_intake.py --query "breast cancer" --limit 10 --sort downloads \
    --report data_candidates.json
python scripts/dataset_intake.py --inspect scikit-learn/breast-cancer-wisconsin

Output light.data_candidates.v1, including raw_response_sha256 and candidate_manifest_sha256; missing license/revision/last_modified/size/split/file tree/card, license only from tag, gated/private, or hitting medical/privacy/human and other sensitive tags → review, API failure → UNAVAILABLE exit 2。It does not produce light.findings.v1, and does not enter STAGE_GATES; for the full closed loop see references/data-resource-map.md。

① Data feasibility precondition gate → if insufficient, 2⊣3 (blocked before the idea, one of the critical souls of this skill)

# 规模够不够支撑 idea 所需统计功效（经验粗筛）+ 四问 → critical/warn findings：
python scripts/data_feasibility_gate.py --spec feasibility_spec.json --report feas_findings.json   # insufficient → exit 1
# 交总控聚合（stage 2 确认点，critical fail → exit 1 确定性阻断）：
python ../light-orchestrator/scripts/run_checkpoint.py --file .light/passport.yaml --stage 2 \
    --findings feas_findings.json --write --ts 2026-06-18T11:00
# fail → 根因回炉建议（命中 ROUTES[2]，建议 2⊣3：拦在 idea 前，只建议不执行，停下问用户）：
python ../light-orchestrator/scripts/reroute.py --findings feas_findings.json --stage 2 \
    --passport .light/passport.yaml
# 用户拍板回炉后落账：
python ../light-orchestrator/scripts/passport.py add-back-edge --to 3 --from 2 \
    --root-cause "数据不足以支撑 idea 所需统计功效" --evidence-ptr "<reroute 给的指针>"

feasibility_spec.json：{project, idea, sample{task,n,classes,features,positives,per_class}, feasibility{sufficiency, quality,feature_value}}(scale defaults to being auto-filled by sample). The spec comes from the "data feasibility required-answer fields" of the idea-generation project charter.

② Data leakage critical gate (the second critical soul of this skill)

# 四类泄漏审计 → leak_findings.json（HIGH=critical）：
python scripts/split_leakage.py --train train.csv --test test.csv --group-col user_id --target y \
    --out leak_audit.md --findings leak_findings.json          # 任一 HIGH → exit 1
# 单文件带 split 列：--csv data.csv --split-col split
# 交总控聚合（critical fail → exit 1 阻断；泄漏在 stage 2 内修复，非跨阶段回边）：
python ../light-orchestrator/scripts/run_checkpoint.py --file .light/passport.yaml --stage 2 \
    --findings leak_findings.json --write --ts 2026-06-18T11:30

③ Leakage-prevention splitting / quality gate / health check / drift (orchestrated, can also be run standalone)

python scripts/data_doctor.py --csv data.csv --target y --out report.md     # 体检画像（先做）
python scripts/safe_split.py --csv data.csv --target y --task group --group-col user_id --group-clf  # Pipeline+CV 折内 refit
python scripts/quality_gate.py --csv data.csv --rules rules.yaml --out gate.md   # YAML 数据门禁（CI）
python scripts/sample_size_check.py --task clf --n 1200 --classes 3 --features 20  # 经验功效粗筛（非 power analysis）
python scripts/data_feasibility.py --project X --q1 ok:... --scale-json size.json --q4 ok:... --out data_feasibility.md
python scripts/drift_check.py --ref train.csv --cur test.csv --out drift.md  # KS+PSI（PSI 为主 p 为辅）
python scripts/check_access_level.py --level raw --sink paper   # raw→public 阻断
python scripts/croissant_export.py --in card_fields.json --out ds.croissant.json   # Croissant 元数据
python scripts/derive_eval_set.py --base data.csv --spec derive_spec.json --outdir derived/  # research-plan 回边

Each script --selftest/--help is the interface; for the resource closed loop see data-resource-map.md, for per-tool API/known pitfalls see references.md。

Academician-level deep dive: the four are the passing line (blueprint §4.3-2, not bonus points)

① Data feasibility precondition (many ideas die because the data is simply insufficient/unobtainable/poor quality)

Before proposing the idea beforeask four questions first: what data does this idea need?Is the scale/quality/annotation enough to support statistical significance？sample_size_check for empirical rough screening (classification each class ≥50 is tight/≥100 is more stable; regression EPV samples/features ≥10/≥20; binary classification positive cases EPV, Peduzzi 1996),data_feasibility the four questions take the worst tier.Data idea-killing insufficiency = critical precondition gate，reroute suggested 2⊣3(blocked before the idea, supplement data/change idea) — do not let a idea that the data cannot support stake months on it.

② Data leakage precondition check (a high-frequency landmine for top-conference rejections, critical veto)

Kapoor-Narayanan (Patterns 2023, 2207.07048) survey found 329 papers in 17 fieldsconcluded over-optimistically due to leakage, giving 8 types of leakage. This skill checks the several machine-checkable types (benchmarked against Deepchecks TrainTestSamplesMix/DateTrainTestLeakage*/IndexTrainTestLeakage）：

  • standardization/preprocessing before splitting(fit crossing over) → safe_split in-fold refit eliminates it.
  • time-series crossing(using the future to predict the past) → TimeSeriesSplit + --time-col ascending-order check.
  • train-test entity overlap(same user/patient/pasture across splits) → split_leakage --group-col，GroupKFold prevention.
  • target encoding leakage(target mean encoding used the full set including test) → split_leakage --target check signature. Any HIGH→critical→exit 1. This is stage 2's STAGE_GATES (leakage).

③ Judgment of the data's minable value (is there any structure/pattern worth working on)

data_doctor profile + four questions Q4: are the feature-target relationships real (not ID-like misuse, not target leakage), is there any modelable structure. data-centric perspective (DataPerf): changing the data sometimes beats piling on models.This is a qualitative judgment + profile heuristic, not a comparable score。

④ Self-built dataset (annotation spec / privacy compliance / publishability / license)

templates/annotation_guide.md(category definitions/boundary rules/LLM-assisted + human review loop/QC sampling rate/IAA) + assets/data_card_template.md (aligned with Datasheets for Datasets / HF Dataset Card / Croissant: motivation/composition/collection/annotation/uses/distribution/maintenance + bias·privacy·access tiering·provenance). Annotation quality: IAA (sklearn cohen_kappa_score / statsmodels fleiss_kappa) evaluates the process as a whole + cleanlab confident learning locates specific suspicious samples (human adjudicates top-K, not all auto-deleted). Privacy/license compliance must be reviewed manually and by research-ethics.

Closing self-check (go through before issuing verdict / before writing back to the master control)

  • candidate data's source/revision/license locator/gating/size/split/file tree/last_modified verified today? dataset_intake does the report have raw_response_sha256 and candidate_manifest_sha256 ? Was the URL recorded before sampling, and the SHA256 before the whole database?
  • ran data_identity_fitness.py ? Do data identity, permission chain, derivation chain, split threat matrix, fitness and staleness all have locator/hash/impact?
  • as_of、snapshot_at、split created_at、data_valid_at is it a real verification/freeze time? Didn't prefill a future date, right?
  • didn't use downloads/likes/usability to substitute for task match and representativeness judgment, right? For restricted or license unknown, truthfully write review/unresolved , right?
  • are the raw→clean hash, transform script/commit, row mapping and split scheme/seed/group/time all kept?
  • does DERIVED data have per-step input/output SHA256, transform locator, commit? Permission UNKNOWN/RESTRICTED/PROHIBITED wasn't written as "usable", right?
  • before proposing an idea, judgeddata feasibility? For those with insufficient data idea-killing,stop and use ASK to ask the user 2⊣3 back-to-the-drawing-board decision made? (didn't arbitrarily let it pass/send it back)
  • all fit-type preprocessing put into Pipeline fit on training folds only ? Didn't fit_transform the full set before splitting, right?
  • time-series data uses TimeSeriesSplit didn't shuffle it into a mess, right? Repeated individuals use GroupKFold to prevent entity overlap, right?
  • ran split_leakage checkedfour types of leakage? Is a HIGH hit real contamination or suspected legitimate (stop and ask the user)?
  • didn't treat empirical power rough screening as power analysis, didn't hype heuristic leakage detection into "checked everything" , right? (honestly mark the boundaries)
  • augmentation/resamplingonly on training foldsdone? Didn't contaminate validation/test folds, right?
  • raw/unmasked data didn't flow to any public link, right (check_access_level)? Data sources are verifiable, no fabricated DOI, right?
  • cleanlab's so-called "mislabeled" ishuman-adjudicated top-K, not fully automated deletion, right?

Name-reality alignment (honest, not hyped as a selling point)

True increments (v2 delivered, already selftest)：⓪ Unified contract for data identity/permissions/lineage/split threat/fitness（data_identity_fitness.py, new in Round 3) — produce light.findings.v1, aligning license/consent/DUA/ethics, raw SHA256, derivation chain, split threat matrix, staleness impact with FIT/FIT_WITH_LIMITATIONS/NOT_FIT/UNKNOWN ; UNKNOWN/prohibited/unexcluded threats, future timestamps, andNOT_FIT/UNKNOWN adjudications will not be passed off as pass. ① Data candidate intake before download（dataset_intake.py, new in Round 2/continued in Round 3) — HF public API metadata normalized to light.data_candidates.v1, verifying access/license/revision/last_modified/size/split/file tree/card, recording raw_response_sha256 and candidate_manifest_sha256; missing fields, tag-only license, sensitive tags, or gated/private are honestly review, not passed off as usable, and the critical surface is not expanded. ② Data leakage critical gate producer（split_leakage.py port v1,v2 fixes hardcoding ../../_shared→canonical bootstrap + producer m02→data-engineering) — four leakage categories → leak_findings.json（light.findings.v1, HIGH=critical), aggregated by run_checkpoint --stage 2 into exit 1; the output name is exactly the standard artifact referenced by STAGE_GATES[2] . ③ Data feasibility pre-critical gate producer（data_feasibility_gate.py，v2 net-new wiring) — the orchestration port's sample_size_check+data_feasibility (v1 pure tool, zero integration _shared, grep-verified) produces critical/warn findings, insufficient → reroute hits ROUTES[2] suggests 2⊣3 (blocked before idea). ④ Pipeline+CV preventing fit crossing（safe_split in-fold refit assertion). ⑤ Zero-heavy-dependency data gate(quality_gate is a lightweight isomorphic implementation of GX philosophy, pure pandas+PyYAML).

What a bare model can already do (not hyped): "split data before standardizing" "watch out for leakage" "sample size must be sufficient" "datasets need a card" — bare Opus can do all of these. This skill's value = ① Turning leakage prevention into a deterministic machine-readable gate + in-fold refit assertion(a bare model says no leakage verbally but still fits on the full set); ② Data feasibility placed before idea finalization + 2⊣3 back edge(a bare model won't "block before idea" and feed back to the idea stage); ③ Machine-readable critical findings + deterministic blocking + root-cause recycling (a bare model gives verbal conclusions, which the orchestrator cannot read or block on).

Honest shortfalls (known not done)：

  1. Auto-discovery currently only integrates HF public metadata: OpenML/UCI/Zenodo/Kaggle use resource maps + harness networking;metadata-ready only means fields are relatively complete, not that the task fits, that licensing is finally adjudicated, or that there is no privacy issue/leakage. When the HF API fails, exit 2 + UNAVAILABLE, not passing off old cache as the day's true value.
  2. Leakage detection is heuristic and bounded：split_leakage checks several categories ofsignatures(exact/near-duplicate/entity/target-encoding crossing),≠"checked all leakage"; near-duplicates rely on binning fingerprints (coincidences cause false positives, MED labeling requires manual review); target encoding leakage's "legitimate group statistics" may also hit (must manually verify the source). Kapoor's "test set not from target distribution / sampling bias / use of illegitimate features" mostly requires manual judgment and can't be covered by scripts.
  3. Statistical power is an empirical threshold, not power analysis：sample_size_check is a coarse screening based on domain empirical lower bounds (per-class samples/EPV/detection instances),does not replace formal argumentation of effect size + significance + power (statsmodels/G*Power). Thresholds are empirical defaults and adjustable.
  4. The feasibility four-question tier is a human/script judgment, not automatic ground truth：data_feasibility_gate onlyaggregates criteria + outputs findings, and doesn't decide for you whether "the data is actually enough"—GIGO (if the input four questions/scale parameters are wrong, the conclusion is wrong).
  5. Doesn't run confident learning / doesn't bundle cleanlab, doesn't do a full EDA suite: annotation quality relies on specs + IAA + (external) cleanlab;data_doctor is coarse profiling, not complete EDA; drift/quality gates use lightweight self-written implementations (drift_check pure numpy asymptotic p, quality_gate without heavy GX dependencies), expressiveness is inferior to Deepchecks/GX/ ydata-profiling; for heavy scenarios still use professional libraries (references has real endpoints).
  6. Doesn't decide privacy/license compliance for you + doesn't do full Croissant/datasheet validation：check_access_level judges flow direction based on declarations (true de-identification requires manual work + research-ethics); croissant_export outputs the critical layer (full spec validation/Hub upload left to external tools).
  7. v1 asset trade-offs (honest): ported split_leakage/safe_split/sample_size_check/data_feasibility/quality_gate/data_doctor/ drift_check/check_access_level/croissant_export/derive_eval_set;emit_artifacts.py not ported—its "standard artifact names + passport registration" goes to memory-pm in v2 pm.py / orchestrator passport.py append-stage, not rebuilt (standard artifact name conventions see this SKILL's "Outputs"). v1's code_assets/ shared statistics library (stats_tests/agreement) not ported in v2, statistics/κ done directly with statsmodels/sklearn.

Standard output artifacts:data_feasibility.md(handed to idea-generation/idea-critique, prerequisite 2⊣3) · leak_findings.json(leakage critical gate) · quality_report.md / data_card.md(handed to research-plan/experiment-coding for experiments). Written to .light/, passport registration handed to memory-pm.

References (three-level progressive disclosure: read when needed)