SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-project-structure

light-project-structure

Data & analytics

View repository

Updated 10-16 00:00

About this skill

Standardized project folder organization

Standard research project skeleton

project-name/
├── data/            原始/中间/处理后数据（大文件走 DVC）
│   ├── raw/ interim/ processed/ external/
├── src/             源代码
├── models/          模型权重与定义
├── results/         实验结果（指标、输出文件）
├── figures/         论文/报告用图
├── docs/            项目文档
├── paper/           论文（LaTeX/Word 工程）
├── ppt/             演示文稿
├── patent/          专利材料
├── software-copyright/  软著材料
├── experiments/     实验脚本与配置（按实验编号）
├── logs/            运行日志（MLflow/W&B 本地）
├── configs/         配置（Hydra yaml）
├── references/      文献库（Zotero 导出/.bib）
├── assets/          图标/模板/素材
├── notebooks/       探索性分析
├── .light/          编排台账 passport.yaml + handoff/ 会话衔接卡（纳入版本控制）
├── README.md  CHANGELOG.md  .gitignore  pyproject.toml/environment.yml

Data layering follows Cookiecutter Data Science:raw/(raw immutable) interim/(interim) processed/(final for modeling) external/(third-party sources). notebooks are named "number-abbreviation-description" (e.g. 1.0-jl-eda.ipynb）。

The data flow is the DAG (core method)

Think of the entire project as adirected acyclic graph: nodes are data/artifacts, edges are deterministic transformation scripts. Organize directories along three iron rules:

  1. raw is immutable：data/raw/ Once it lands, it is never modified in place and never written back to by scripts. Any cleaning/transformation reads raw and writes interim/ or processed/. In this way the original facts are always traceable, and the DAG has a trustworthy source.
  2. Every artifact can be recomputed from upstream：processed/、models/、results/、figures/ are all downstream nodes of the DAG, and even if deleted they can be regenerated from "upstream data + code + params"—therefore they go into .gitignore/DVC, not the Git text repository. What can be recomputed does not enter the repository; this is a direct corollary of the DAG approach.
  3. notebooks split into exploration vs reporting：notebooks/exploratory/(drafts, can be messy, numbered naming 1.0-jl-eda.ipynb) and notebooks/reports/(clean, rerunnable, produce external charts) are separated. Exploratory notebooks do not enter the DAG critical path; finalized logic should sink down into src/, imported jointly by notebooks and pipelines, avoiding "logic living only in some cell".

Use dvc.yaml to explicitly declare this DAG (a stage's deps→outs are the graph's edges),dvc repro accordingly reruns only the affected downstream nodes;dvc dag can print the dependency graph.

Naming conventions

  • Directories/files use lowercase + hyphens or underscores, no spaces or Chinese.
  • Experiments/versions carry a date or sequence number:exp_004_ablation, paper_v3, fig3_ablation_v2。
  • Data files include version/split markers; result file names map to experiment numbers and are traceable.

Organizing actions

  1. Inventory the existing project (borrowing the repo-intake-and-plan approach): first read the README→scan setup scripts and documented commands→classify the workflow as inference/training/evaluation, then use that to put scattered files in their places.
  2. New project skeleton:Preferred scripts/scaffold.py—one command builds the whole tree + copies templates + optional --dvc/--uv|--poetry： python scripts/scaffold.py ./my-proj --name my-proj --dvc(generates the four data layers + notebooks exploration/report split + src package + .light/(passport ledger + handoff/ handoff card directory, included in version control) + lands 6 templates + pyproject.toml; a non-empty target requires --force). pyproject defaults to uv backend(consistent with the a03 recommendation), add --poetry to switch to the Poetry alternative. Or use CCDS (pipx install cookiecutter-data-science then ccds); or generate manually according to the skeleton above. Either way, README/CHANGELOG/PROJECT_PLAN/.gitignore/.editorconfig/pyproject.toml must be landed.
  3. Rename and standardize→add README/CHANGELOG→produce a "file placement note" (borrowing the handoff structure: Key Decisions table + failed attempts + warnings).

Versions and dependencies

  • Git manages text, DVC manages large files:dvc init→dvc add data/x.csv(generates .dvc small files and automatically writes .gitignore) →git add *.dvc .gitignore→dvc remote add -d storage s3://...→dvc push. Pull:git pull && dvc pull。.dvc/dvc.yaml/dvc.lock/.dvc/config into Git, the data itself goes into DVC.
  • Use multi-stage pipelines with dvc.yaml(stage field:cmd required,deps、params、outs、metrics、plots)，dvc repro re-runs only changed stages (frozen:true stages skipped); experiments use dvc exp run/show/diff。
  • Dependency lockingdefault uv(consistent with a03 backend-coding, fast and lock is cross-platform):uv init→uv add→uv sync(installs exactly per uv.lock ); dev/test dependencies uv add --dev go into [dependency-groups]; application projects commit uv.lock。alternative Poetry：poetry init→poetry add→poetry install, dev/test go into [tool.poetry.group.dev.dependencies], commit poetry.lock. The scaffold outputs the uv version of pyproject by default;--poetry switches to the alternative.

Quality gates (configuration essentials)

The configurations of the four-piece set—Ruff (replacing flake8+black+isort), pre-commit (rev pinned to tag, hooks into ruff-pre-commit), EditorConfig (root=true to unify indentation/line endings), .gitignore (GitHub Python template + research entries)—have been placed into the templates/ corresponding templates; for each item, see [tool.*] keys and commands in references.md; the quality gate criteria are consistent with a03 backend-coding (a03 is the single source of truth on the code side). Choose one of two task runners: if make already exists, use Makefile; for cross-platform + readability use Taskfile (checksum skip,task -l）。

Corresponds to the project repository

The directory structure corresponds one-to-one with the status fields of db09 project_card (paper_status↔paper/, code_status↔src/, data_status↔data/…), making it easy for a02 to track progress.

Ready-made templates (in this skill directory templates/, can be copied and used directly)

  • scripts/scaffold.py — Generate the full skeleton with one command: create the directory tree + copy 6 templates to the project root (remove the .template suffix) + write src/<module>/__init__.py + always drop pyproject.toml(default uv backend,--poetry switches to the alternative, both include Ruff);--dvc additionally write dvc.yaml，--force overwriting if non-empty. Both paths have been tested and pass selftest.
  • 6 templates:PROJECT_STRUCTURE.md(directory conventions + naming + db09 correspondence),README.template.md、PROJECT_PLAN.template.md(checkable tasks + No-Placeholders),CHANGELOG.template.md（Keep a Changelog+SemVer）、python-research.gitignore(baseline taken from the official GitHub Python.gitignore, HTTP 200@2026-06-06, with research entries added),editorconfig.template、pre-commit-config.template.yaml(rev pinned to tag, maintained bypre-commit autoupdate --freeze , takes effect only after pre-commit install inside a git repository). For the purpose of each template and configuration details, see references.md。

When creating a new project, prefer python scripts/scaffold.py <dir> [--poetry --dvc] to get it done in one step; or manually copy the above files to the project root (remove the template suffix from gitignore/editorconfig), and fill in {{占位符}} in README/CHANGELOG/PROJECT_PLAN.

Outputs

A regularized directory tree + the above templates/ README/CHANGELOG/.gitignore/.editorconfig landed files + a "file placement notes".

Connection

Provide a landing structure for m05/a03/a04; reserve corresponding directories for m15 (software copyright/patent), m07 (papers), and m16 (PPT); sync status with a02/db09. When the organization workload is large, break it down according to "independently completable + with acceptance criteria + marked dependency order" (borrowing the vertical slicing approach of to-issues: each task is an end-to-end path that can be independently claimed, rather than sliced horizontally by layer); for complex organization, first produce a plan of "precise path + specific actions + verification steps" before executing, written to the extent that someone without context can follow it (borrowing writing-plans, with task granularity of 2–5 minutes). For reproduction/audit scenarios, fix repro_outputs/ Archive the results of each run, recording the three-state markers verified/partial/blocked + output + PATCHES.md(which files were changed) + a list of assumptions and blockers (borrowing minimal-run-and-audit).

For verifiable details of tools and methods (real commands/configurations/endpoints/links), see the same directory references.md; for ready-to-copy templates, see the same directory templates/。