SuperAPI AI Directory
Partner中文AI newsFavorites
Skillscosmos3-nano-transfer-auto-inference

cosmos3-nano-transfer-auto-inference

Agents & models

View repository

Updated 10-23 00:10

About this skill

Run inference with Cosmos3-Nano-Transfer-Auto

Generate one video per selected camera from that camera's WSM control and caption. Add an HD-map range control for joint RGB/LiDAR generation. Use initial camera images for image-to-video, or omit them for text-to-video with WSM.

The released model is optimized for the reference camera rig. You can try a custom rig, but expect reduced quality as camera placement, viewing direction, FOV and lens projection diverge. Choose between retargeting WSM to the reference rig and retaining your own rig; the latter may require post-training.

Start here

  1. Try the reference scene first. Use its complete request and the released model to establish a baseline before substituting your own inputs.
  2. Choose and match your cameras. Follow the rig workflow, then use the Camera Rig Explorer and mapping guide to choose the closest supported identities for your rig. Keep each camera's RGB, WSM and caption together; camera_key selects its learned view embedding.
  3. Prepare camera-local captions. Describe the requested scene in the training-aligned caption style, then check synchronization and agreement across views.
  4. Generate, inspect, then tune. The reference recipe uses 720p, guidance 3, shift 10, 35 steps and the bundled negative prompt. Read what these settings do before changing them; inspect every camera, not only the front view.

Choose the task

  • Check inputs or existing captions: read input preparation and complete only the requested checks. Existing-caption checks do not invoke authoring or rewrite the original.
  • Generate or upsample captions (av-per-view-caption): read caption guide, including its caption policy. Follow its options, defaults and WSM-inspection requirements; finish after the requested caption deliverables.
  • Explain options or defaults: read the relevant guide and explain them without requiring media or starting work.
  • Run generation: follow Prepare, Run, and Sampling and output checks below.

For model/configuration questions, read the model contract. An explanation request does not authorize jobs, uploads or changes to source assets.

Prepare

  1. Follow environment and snapshots. Set TRANSFER_AUTO_HF to a complete local Transfer-Auto snapshot.
  2. Download and checksum-verify the shared reference scene, or prepare your own scene.
  3. Run input preparation, which invokes per-view captioning when requested.
  4. For the reference scene, set AUTO_REQUEST="$AUTO_SAMPLE_ROOT/input-transfer-auto.jsonl". It selects paragraph captions by default; follow Choose a caption style to use the recommended timestamped alternative. For your own data, prepare a Framework JSON/JSONL request using the input and sampling guide. Set "guidance": 3 and "shift": 10 in the request, as the supplied sample and notebook do; these settings are read from JSON in inference mode. Set AUTO_OUTPUT to a fresh output directory.

The prepared artifact is a request plus camera-local caption sidecars and synchronized controls/optional initial observations. A camera-to-text dictionary alone is not executable.

Supply scene-caption bodies only. Framework automatically adds the selected camera rig and current view's direction/lens/FOV description, plus the WSM-control task instruction as a system prompt. Do not duplicate either in your caption files. See automatic prefixes.

Run

From the Framework root, inside your GPU allocation:

export COSMOS_INTERNAL=0 COSMOS_TRAINING=0
AUTO_NEGATIVE_PROMPT="$(cat .agents/skills/cosmos3-nano-transfer-auto/inference/assets/negative-prompt.txt)"
torchrun --standalone --nproc-per-node=4 \
  -m cosmos_framework.scripts.inference \
  -i "${AUTO_REQUEST:?set a prepared request}" \
  -o "${AUTO_OUTPUT:?set a fresh output directory}" \
  --checkpoint-path "${TRANSFER_AUTO_HF:?set the local snapshot}" \
  --resolution 720 --num-steps 35 \
  --per-view-negative-prompt "${AUTO_NEGATIVE_PROMPT:?Run from the Framework root and check the bundled negative-prompt.txt}" \
  --cp-size 4 --cfgp-size 1 \
  --no-use-torch-compile --no-diffusion-cache

The request supplies guidance 3 and shift 10; the command sets 720p, 35 steps and the same bundled negative text as the native Framework notebook. Only these CLI options override the request; the downloaded JSONL stays unchanged. Cameras, captions, controls, initial images, FPS, frame count and seed also come from the request.

Use explicit physical views in the request; do not add a canonical-view-count override when preserving a custom subset. Model snapshots contain their own configuration; no training experiment selector is needed for this command. The distributed example requires four suitable GPUs; memory fit and speed depend on the request and hardware. Read GPU memory and parallelism before changing the allocation. Keep the installed runtime's guardrail policy; any benchmark-specific disabling must be explicit.

Sampling and output checks

The reference request generates 297 frames at 720p/30 FPS. For a short first check, change the command to --resolution 480 and prepare a 61-frame interval. Match caption spans and control coverage to the selected interval: caption-driven Transfer-Auto output length can differ from a top-level override.

Both 480p and 720p are supported. Use aspect_ratio: "16,9" for the training-matched 832×480 or 1280×720 sizes; see resolution and aspect ratio. Transfer-Auto trains on approximately 13.3-second chunks and supports longer RGB sequences through chunkwise continuation. Read the duration guide before extending a request, especially for joint RGB/LiDAR.

Check source timing, model FPS and playback FPS separately. The reference camera rate is 30 FPS; changing --fps does not resample controls, and declaring 10-FPS frames as 30 FPS is only a diagnostic experiment, not a quality fix.

For separate per-view captions, route the supplied negative text through per_view_negative_prompt, not negative_prompt. Preserve the user's explicit settings in comparisons. The sampling guide explains the guidance choices and joint LiDAR timing.

Inspect sample_args.json, sample_outputs.json, all camera/control videos, and lidar.safetensors for joint output. Check view identity, actual duration, numeric range/intensity and frame IDs. Save 50/50 WSM/RGB overlays using matching crops and times. A clean overlay checks geometry, not the correctness of a physical camera alias.

Use a new output directory when changing inputs or settings.

References