cdf-auto-labeling

CDF Auto-Labeling

Use this skill when a user wants to kick off CDF Auto-Labeling on their own data, domain, or use case, or when the request matches a shipped cookbook, stage, authoring, or migration task. This is a router: sequence the specialized references instead of duplicating their detail.

Routing (Read First)

Request looks like	Read
New user, clean checkout, first validated run, "how do I get started"	This file, then the matching reference below
Choose annotation targets / stage subset for a domain	references/scenario-planning.md
Create, review, or adapt a cookbook	references/cookbook-authoring.md
Write or adapt VLM/LLM prompts or question banks	references/prompt-authoring.md
Migrate an existing annotation repo into this one	references/pipeline-migration.md
Run the video data augmentation cookbook	references/video-data-augmentation.md
Run or choose an EPAS / PAS cookbook	references/event-and-person-attribute-search.md
Run event-verification reasoning	references/event-verification-reasoning.md
Debug an already-integrated workflow	references/workflow-runner-debugging.md
Pull prebuilt stage images from NGC, log in to nvcr.io, or fix a missing cdf-* image	references/ngc-containers.md
Implement or review a new stage or Dockerized service	references/workflow-stage-integration.md
Configure or debug one production stage	The matching file under references/stages/

Stage references: super-resolution, detection-and-tracking, captioning, visual-qa, reasoning, person-attribute-search, grounding-2d, referring-expressions, training-export.

Instructions

1. Confirm the critical run inputs with the user before doing anything else, and ask a concise question whenever one is missing or ambiguous - never guess or silently invent a default. At minimum confirm: input data path, output path, VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for reasoning-capable models) the max_tokens cap. Restate the confirmed values back to the user before the first execution.
2. Verify the environment: repository cloned, make targets available, the model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is available. State any missing prerequisite as a blocker instead of assuming it. If the stage images are not present locally, offer to pull the prebuilt images from NGC with make pull instead of building them. Use ngc-containers.
3. Run a shipped example first to confirm the stack works end to end before customizing. Pick the closest operator pipeline - video data augmentation, event-and-person-attribute-search, or event-verification-reasoning - and run its committed cookbook. Use the matching operator reference.
4. Plan the target scenario: define modality, domain, intended consumer, and required annotations, and get a minimal stage subset. Use scenario-planning.
5. Adapt the closest shipped cookbook to the new domain rather than authoring from scratch. Use cookbook-authoring.
6. Author the domain prompts and question banks. Use prompt-authoring.
7. Configure the per-stage settings for the domain (detector classes or SAM3 prompts, endpoints, windowing, max_tokens). Use the relevant stage reference, starting with detection-and-tracking.
8. Dry-run the adapted cookbook, then execute and validate the outputs. Use workflow-runner-debugging.

Adopting an existing external annotation or dataset-generation repository into CDF instead of starting from a shipped cookbook is a migration task; use pipeline-migration for that path.

Examples

New user, new domain: "I cloned the repo and have my own warehouse-safety video. How do I produce auto-labels for my domain?"

Guided path:

• Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a shipped example before customizing:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'

• Plan the domain (scenario-planning) -> subset detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export (add grounding_2d for caption→boxes or referring_expressions for boxes→phrases; use grounding-2d / referring-expressions).
• Copy the closest cookbook to cookbooks/warehouse_safety/configs/pipeline.yaml and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.
• Dry-run the new cookbook, then run for real and validate outputs:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'

Guardrails

• Do not guess or fabricate the critical inputs enumerated in step 1; if any is missing or ambiguous, ask the user and confirm before executing.
• Do not customize a cookbook before a shipped example runs clean; a broken base makes domain debugging ambiguous.
• Keep the first custom pipeline minimal - only the stages needed for the requested annotations - and expand later.
• Verify that every selected stage's service package and image exist in the current branch before promising an end-to-end run.
• Do not put secrets, tokens, or absolute home paths in committed cookbooks; use placeholders such as <model-cache> and env vars for endpoint keys.
• For reasoning-capable models (for example Gemini 3 Flash), raise max_tokens on the visual_qa and reasoning LLM substages to avoid the thinking-token tax; keep the default cap for non-reasoning models.
• Do not rely on non-CDF pipelines, commands, or file locations. A first run must be reproducible through workflow-runner:main inside this repo.