vignette-catalog-scaffold

Scaffold a vignette catalog

A catalog starts as a manifest and one useful notebook, not a package or pipeline. Ask for the catalog name, target path, and data surface (rest, duckdb, pooch, or files) only when they cannot be inferred.

Workflow

1. Inspect the target. For a new catalog, require an absent or empty directory. For an existing repository, preserve its files and use --adopt. Ask once which optional conventions to use, unless the user or existing repository policy already specifies the choices: standard data layout and working agreement, output summaries with an index notebook, and committed session snapshots. Explain the options in step 5; allow any combination or none, and wait for the answer before applying them.

2. Generate the minimum structure:

<skill-dir>/scripts/scaffold.py <target> --name <name> --surface <surface>
# Existing repository:
<skill-dir>/scripts/scaffold.py <target> --name <name> --surface <surface> --adopt

The script copies real assets rather than asking the model to reproduce templates from prose.

1. Replace the deliberate NotImplementedError in notebooks/nb01_orientation.py with one bounded call that reaches the actual dataset. Put reusable access logic in a top-level @app.function, and declare its dependencies and versions in the notebook's PEP 723 block.

2. Fill the dataset-specific fields in catalog.toml: description, upstream version, cache, caveats, auth, helper names, and what the orientation notebook demonstrates. Keep dataset identifiers, API behavior, hashes, and operational caveats in this catalog rather than the shared skills.

3. Reconcile existing README.md, AGENTS.md, .gitignore, and pyproject.toml instead of overwriting them when adopting. The generated AGENTS.md contains the exact post-clone skill install commands. Record the selected and declined conventions in its dataset contract, including paths and snapshot policy, and adjust .gitignore to match.

   • If selected, use the standard data layout and working agreement: protected inputs in data/raw/ and data/external/, reproducible shared preparation outputs in data/interim/, and each analysis's workspace in data/processed/<analysis>/. Data flows from raw/external to interim to processed; analyses may also read inputs directly. Never hand-edit inputs or shared intermediates; change the preparation notebook or helper and regenerate intermediates. Create directories only when needed. Record the working agreement in the catalog's AGENTS.md, linking existing manifest entries or documentation rather than duplicating them. Name the input sources, versions or hashes where available, and retrieval helpers or commands. Name the notebook or command that rebuilds each shared intermediate, or state that none are needed; identify each analysis's input and output paths. Explain how collaborators retrieve and share outputs, or reproduce them locally, and who coordinates shared input or preparation changes when applicable. Resolve only missing facts with the user, and verify the documented retrieval and rebuild route on a bounded example before finishing setup. Sharing must preserve other collaborators' outputs and local-only work; setup does not itself authorize uploads or shared-data replacement.
   • If selected, add output summaries and an index: each analysis with saved outputs writes a summary.json containing description, numbers (a name/value object), and files (objects with repository-relative path and caption). Use data/processed/<analysis>/summary.json with the standard layout; otherwise record the chosen output path. Create notebooks/index.py to discover and display these envelopes, tolerating an empty collection. If outputs are ignored, allow-list the envelopes and their parent directories so a fresh checkout can discover them; handle unavailable output files without failing.
   • If selected, commit session snapshots: track marimo snapshots in notebooks/__marimo__/ for output review or browsing, and refresh them during validation with --write. Otherwise keep snapshots ignored.

4. Install the project skills using that documented command, open the orientation notebook through vignette-catalog-compose-notebook, inspect its real output, and run its scripts/validate-notebook.sh.

Use existing catalog helpers and notebooks for retrieval and preparation. Add src/, a shared environment, a workflow engine, or cloud sync only when the project's requirements call for it.