cosmos3-nano-sim-bimanual-troubleshooting

Troubleshoot Cosmos3-Nano-Sim-Bimanual

Find the first broken boundary in setup, data preparation, training, checkpoint loading, generation, or scoring. Start with the cheapest check that can distinguish the bounded causes. Preserve the model contract during recovery.

Find the symptom

Symptom	Likely boundary	Procedure
Import, CUDA, or model access fails	Environment or allocation	Check runtime and access
Width, mask, timing, caption, or split is wrong	Data materialization	Validate one input
Worker cannot import Framework or read shared assets	Launch environment and placement	Inspect staged launch
First batch stalls or iteration 3 cannot load compiled packages	Decode/materialization or shared output root	Inspect training progress
Missing EMA keys, inherited iteration, or wrong teacher	Checkpoint operation and stage roles	Check checkpoint identity and state
Missing input media, mutated prompt, static video, or mesh mismatch	Input access or generation contract	Check generation
Missing metric rows or incomparable PSNR	Population and target alignment	Check evaluation

1. Check runtime and access

Preserve the exact command, code revision, resolved config, immutable checkpoint, selected variant, first traceback, and failing input identity.

From the checkout root with the locked environment active, inspect the package and CLI. A container package outside the staged checkout is a different runtime.

: "${FRAMEWORK_DIR:?set the Framework checkout root}"
cd "$FRAMEWORK_DIR"
source .venv/bin/activate
python -c 'import cosmos_framework, torch; print(cosmos_framework.__file__, torch.__version__, torch.version.cuda, torch.cuda.is_available())'
python -m cosmos_framework.scripts.inference --help
df -h /dev/shm

Keep COSMOS_TRAINING=0 for an inference-only installation; training uses COSMOS_TRAINING=1 and the training extra.

If uv sync --locked reports a stale lock before installation: verify it with the required resolver from the checkout root:
```
uvx --from 'uv==0.11.29' uv lock --check
```
If this passes, the original command used an incompatible uv; repeat setup through the documented uvx command in the environment guide. Do not regenerate the lock with the incompatible resolver. If it fails in a clean checkout at the selected Framework commit, preserve the output and report the revision.
If imports fail: repair the locked environment using the setup guide. If CUDA fails: repeat GPU checks inside the target allocation before replacing dependencies.
Check local checkpoint, tokenizer, data and media readability. HF downloads may need model read access. Require provider credentials only for resources that are actually remote. Never copy or print credentials. Continue when imports, help, allocation CUDA, and selected access checks pass. Then rerun the original boundary with one sample.

2. Validate one input

For a schema-5 input, use Framework's checker against the actual export and the smallest failing manifest:

: "${CHECKPOINT_DIR:?set the complete local schema-5 HF export}"
: "${INPUT_JSONL:?set the failing schema-5 manifest path}"
python tools/check_bimanual_mixed_inputs.py \
  --checkpoint-path "$CHECKPOINT_DIR" --input "$INPUT_JSONL"

If width, mask, or domain fails: trace native calibration, geometry, 59-value mapping, normalization, and 64-value invalid padding through the action schema. Repair the actual mapping. Do not reshape values or relabel an embodiment.
If timing fails: inspect timestamps and resampling. Require T actions for T+1 observations and 4k+1 output frames. Do not trim one modality silently.
If captions or splits fail: match captions to the selected interval and isolate entire episodes. Remove future-outcome leakage when the task conditions only on the initial scene.
Repeat JSON validation, the schema round trip, and loader materialization. Continue with finite tensors, aligned times, grounded captions, and disjoint episodes. For a local training bundle, follow the native data preparation and short training check.

3. Inspect staged launch

Use the native torchrun commands in post-training. Confirm the package origin is the intended Framework checkout on every rank.
Check the environment, complete bundle, local model/VAE assets and output root are readable at the same absolute paths on all workers.
For multi-node runs, follow the Framework launcher notes. Check unique node ranks, one reachable rendezvous address, GPU count and NCCL. Confirm these before retrying the failing stage.
The failure playbook covers shared-path and configuration checks.

4. Inspect training progress

For OOM or collective hangs, inspect world size, parallel degrees, maximum-size sample, token budget, activation checkpointing, worker/prefetch settings, and the first collective error. Change one supported memory setting at a time.
For a stalled first batch, inspect local media decoding, required bundle strata, stage-specific windows and worker progress. Fix the first failing local asset or transform; do not apply remote-loader watchdog settings.
For an iteration-3 zip/AOT load failure, check whether config.job.path_local is node-local. Set an absolute allocation-shared IMAGINAIRE_OUTPUT_ROOT and unique run name. Rerun the config check and restart from the stage input.
For a missing libnppicc.so.13, follow the Framework setup library checks and keep the locked environment. Framework discovers wheel-provided NPP libraries; also check the FFmpeg library search path and available /dev/shm in every target worker.
For non-finite loss, inspect the first offending batch, action ranges, masks, flow timestep, gradients, LR, and frozen-reference inputs. Keep the bad batch for diagnosis. Do not bypass it by silent sample dropping.
Require first-batch completion, finite optimization, an aligned checkpoint save, fresh-process readback, and matched generation after recovery. With compilation enabled, cross iteration 3 and finish a later finite iteration.

5. Check checkpoint identity and state

Identify the stage, exact saved iteration, regular-or-EMA variant, and whether the requested operation is a weight-only warm start or same-stage resume. Inspect the checkpoint contract before changing load strictness.
If a new stage inherited iteration or optimizer state: set checkpoint.load_training_state=False and apply that stage's exact input and skip-list row. Stage 2 uses Stage 1 for both net and frozen teacher. Stage 3 uses Stage 2 generation weights and Stage 0 reference/critic inputs.
If a Stage 0 load from Cosmos3-Nano reports missing net_ema.*: restore strict regular loading, skip the absent Cosmos3-Nano EMA tree and fresh action modules, then copy the completed regular tree into EMA through the source-checked post-load callback. Do not weaken strictness to hide unexplained keys.
If full-state resume is requested: require the same stage and compatible config, model, optimizer, scheduler, EMA, preprocessing, dataset identity, sampler, topology, and variant. Otherwise take a fresh weight-only run.
Use the native local-loader resume callback. Verify restored counters and loader state for resume, or fresh counters for a warm start. Classify every unmatched key and stop on a real architecture mismatch.
If export finds no vision tensors: inspect the source keyspace. The exporter accepts one unambiguous visual.* or model.visual.* prefix and normalizes it. A source containing both is invalid. This is shared exporter behavior, although the mismatch was first observed on Stage 3 output.

6. Check generation

Run the native checkpoint and input check. The checker verifies the export, image, and action sidecar before loading weights. Check decoding in the target environment as well. For user-owned remote media, establish access separately; checkpoint credentials do not authenticate it.
If an object is confirmed absent: choose another supported row or user-owned media. Preserve relative action paths. Do not cancel an independent healthy training run.
For field rejection, compare the imported package origin with the staged source before editing valid input fields. For a short, static, or corrupt video, inspect the actual checkpoint, schema, normalized actions, frame count, domain, prompt, seed, and decode properties.
For a schedule mismatch, select the matched checkpoint and command. The current released schema-5 model and canonical user-trained Stage 3 student both use four steps, but each export owns its exact ladder. Preserve the selected export's saved schedule and matching CLI step count.
For changed punctuation or resolution text, check the export's prompt mode and input manifest. The matched schema-5 export selects plain text. Keep the exact intended prompt and follow the matching native example.
For process-group mismatches, require the resolved world size to equal the launched count. Verify divisibility of DP and CP/CFG products before loading weights. Retest one row before scaling.
Continue only with one playable output per input row after the bundled validator fully decodes it and confirms exact saved frame count, dimensions, frame rate, and spatial/temporal change. Source, padded-canvas, and saved dimensions may differ after spatial-compression alignment. Keep unchanged intended conditioning, a complete metadata record, and Cosmos3-Nano, released, and user-trained outputs labeled with their actual checkpoint identities.

7. Check evaluation

Compare planned sample IDs with output and metric rows before aggregation. Keep failures, skips, duplicates, corruption, and missing targets in the denominator.
For PSNR discrepancies, match membership, target alignment, RGB/action preprocessing, checkpoint variant, seed, generation recipe, and metric implementation. Do not resize, truncate, or drop samples to force a score.
Recompute aggregate counts and PSNR from the saved rows. Continue only when accounting reconciles. Report a recipe comparison if an invariant changes. Follow the evaluation guide for the saved results and comparison procedure.

Return to the workflow

Record the symptom, root cause, correction, and checks rerun.
Return training to post-training, generation to inference, and scoring to evaluation.
State whether existing outputs remain comparable.

About this skill