SuperAPI AI Directory
Partner中文AI newsFavorites
Skillscosmos3-nano-sim-auto-inference

cosmos3-nano-sim-auto-inference

Agents & models

View repository

Updated 10-23 00:09

About this skill

Run inference with Cosmos3-Nano-Sim-Auto

Use the four-step causal student to generate synchronized camera views and optional LiDAR from initial RGB, per-camera captions and WSM/HD-map controls.

The released model is optimized for the reference camera rig. You can try a custom rig, but camera geometry differences can degrade quality. Follow the reference-rig retargeting or custom-rig workflow. Retargeting WSM does not reproject initial RGB images. For adaptation, fine-tune Transfer-Auto on your rig's paired data, then distill it into Sim-Auto.

Start here

  • New rig: compare it with the reference cameras in Camera Rig Explorer, then follow the shared camera-to-embedding mapping. Map each camera to the closest supported identity and keep RGB, WSM and caption assignments consistent. For domain/rig adaptation, post-train Transfer-Auto first and then distill it.
  • Captions: use the same camera-local, training-aligned style as Transfer-Auto, checking appearance and motion across views and time.
  • Sampling: use guidance 1, control guidance 1 and the checkpoint's four-step schedule. Read why Sim-Auto settings differ; guidance 1 still uses captions and controls, but a negative prompt adds no CFG effect.
  • Timing: use aligned 30-FPS cameras for the reference recipe; joint Sim-Auto requires that camera rate. Read the FPS guide before using lower-rate recordings. Changing playback speed does not correct model timing or resample controls.
  • Size and duration: both 480p and 720p are supported; use the training-matched landscape preset. Sim-Auto supports long rolling generation, with qualitatively observed multi-minute rollouts. Read the duration guide for quality expectations and full-horizon input requirements.

Choose the task

  • For input preparation only, follow the input-preparation guide and caption guidance. Return the checked media inventory and per-view caption sidecars without launching generation.
  • To generate videos and optional LiDAR, continue below.

Prepare

  1. Follow Framework setup. Set SIM_AUTO_HF to the downloaded student directory, or a complete student export from distillation.
  2. Follow input preparation. Both Auto models share the reference scene; download the sample and verify its checksums.
  3. Set AUTO_REQUEST="$AUTO_SAMPLE_ROOT/input-sim-auto.jsonl" for the reference scene, or prepare your own JSON/JSONL request using these fields. Set a fresh AUTO_OUTPUT directory. The sample request selects paragraph captions; use caption selection for the recommended timestamped alternative.

Both models use daytime_scene: seven cameras, WSM v4.0.1 controls and paired RGB/LiDAR targets. AUTO_SAMPLE_ROOT points to the downloaded assets/sample_data directory; Sim-Auto uses its input-sim-auto.jsonl request.

Supply scene-caption bodies only. Framework automatically adds the camera-rig and current-view direction/lens/FOV description, plus the WSM-control task instruction as a system prompt. Do not paste these into your caption files; see automatic prefixes.

Use an exported model snapshot, not a training experiment or raw DCP directory. The artifact must include per-view-caption metadata, matching tokenizers and rolling_kv_cache_chunks, rolling_kv_sink_chunks, rolling_prompt_policy. Do not guess missing history settings from the model name.

Run

export COSMOS_INTERNAL=0 COSMOS_TRAINING=0
torchrun --standalone --nproc-per-node=4 \
  -m cosmos_framework.scripts.inference \
  -i "${AUTO_REQUEST:?set a prepared request}" \
  -o "${AUTO_OUTPUT:?set a fresh output directory}" \
  --checkpoint-path "${SIM_AUTO_HF:?set the local student snapshot}" \
  --cp-size 4 --cfgp-size 1 \
  --no-use-torch-compile --no-diffusion-cache

The request selects camera-only versus joint generation by omitting or including lidar.control_path; there is no separate --joint-generation flag here. The fixed student schedule comes from its model configuration. Do not add Transfer-Auto's 35-step/shift/negative-CFG recipe.

The causal frontend runs eagerly with serial CFG. It disables unsupported compilation/CUDA-graph/cache options and reports adjustments. Rolling history comes from checkpoint metadata; older generic KV/sink-size flags are rejected. Memory fit and speed depend on the request and hardware. Read GPU memory and parallelism before changing the allocation; a four-GPU example is not a guarantee of fit on every device.

Check the result

Inspect the sample output directory:

  • vision_view<NN>_<camera_key>.mp4 and corresponding control videos.
  • caption_chunks.json and rolling_prompt.json: text and transitions actually used.
  • sample_args.json and sample_outputs.json: resolved request and completion.
  • Joint only: lidar.safetensors with range_m, intensity and frame_ids, plus decoded lidar.pt.

At 297 camera frames and 30 FPS, joint inference generates 99 sweeps at 10 Hz. Check source IDs and numeric ranges, not only a rendered point cloud. Inspect all camera views and 50/50 WSM overlays; compare caption transitions over the full requested horizon.

For a preview, the Framework provides python -m cosmos_framework.scripts.multiview_collage "$AUTO_OUTPUT" --tile-width 832. Preserve original numeric outputs and videos. Use troubleshooting for failures.

References