isaac-sim-robot-navigation

Isaac Sim Robot Navigation — Runtime

Purpose

Drive mobile robots at runtime with RL policies, trajectory followers, and physics/baked/per-frame stage strategies while avoiding common Kit 110 pitfalls.

Prerequisites

Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
NVIDIA GPU with a current driver (nvidia-smi).
Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Error / symptom	Cause	Solution
Extension or import not found	Wrong `$ISAAC_SIM_DIR` or stale build	Point env vars at `_build/linux-x86_64/release` or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in `isaac-sim-troubleshooting`

Runtime navigation specialization. Driving a robot through a USD scene live, in your own Isaac Sim script.

Read These Skills First

navigation-primitives — occupancy maps, A* planning, differential/holonomic kinematics, robot footprints (Spot Z=0.69, etc.), look-at chase camera math, common gotchas. This skill assumes you know that substrate.
isaac-sim-rendering — RT2, persistent session, ACES, frame validation
isaac-sim-troubleshooting — Kit 110 hang/freeze reference

When To Use This Skill (vs siblings)

Goal	Use
Drive a robot through a scene in real time, see it move	this skill
Record trajectories then re-render with sensors for SDG	`mobility-gen`
Publish/subscribe Nav2 topics to ROS 2	`isaac-sim-ros2-bridge`

Runtime APIs

Capability	API
RL policy execution	`isaacsim.robot.policy.examples.RobotPolicyRunner` + root `get_*_spec` factory
Robot articulation	`isaacsim.core.experimental.prims.Articulation`
Physics lifecycle	`isaacsim.core.simulation_manager.SimulationManager`

For shared primitives (omap, A*, kinematics, look-at), see navigation-primitives.

Physics Simulation on Kit 110 (CRITICAL)

Timeline MUST be playing for Physics simulation

import isaacsim.core.experimental.utils.app as app_utils
app_utils.play(commit=True)  # WITHOUT THIS, PHYSICS NEVER STEPS

Strip non-robot physics from heavy stages

On 33K+ prim stages with multiple physic bodies, simulation hangs. Strip ALL physics from non-robot prims:

from pxr import Usd, UsdPhysics

for p in Usd.PrimRange(stage.GetPseudoRoot()):
    if str(p.GetPath()).startswith("/World/Robot"):
        continue
    if p.HasAPI(UsdPhysics.RigidBodyAPI):
        p.RemoveAPI(UsdPhysics.RigidBodyAPI)
    if p.HasAPI(UsdPhysics.CollisionAPI):
        p.RemoveAPI(UsdPhysics.CollisionAPI)

Kit 110 Exec Model (CRITICAL — 2026-03-15)

Synchronous code executes immediately in --exec scripts. No async needed for scene setup.

asyncio.ensure_future() needs explicit event loop subscription:

omni.kit.app.get_app().get_update_event_stream().create_subscription_to_pop(lambda e: None)
asyncio.ensure_future(main())

capture_viewport_to_file() returns MultiAOVFileCapture — use .wait_for_result(completion_frames=8), NOT .wait().
Pattern: sync scene build → async render scheduling → event loop keeps it alive
Scene load + 100 asset placement + 3-camera render in ~40s total

Camera Chase Cam

The look-at math and standard camera offsets (chase/overhead/POV) live in navigation-primitives. Runtime considerations on top of that:

Camera collision avoidance (TODO): chase camera clips into rack geometry when robot enters narrow aisles. Need raycast from chase eye → robot; if occluded, raise camera or shift laterally. Not yet implemented.

Failure Modes & Recovery

FAILURE: VkResult: ERROR_OUT_OF_DEVICE_MEMORY during navigation render

SYMPTOMS: Vulkan crash partway through capture loop on heavy stage
CAUSE: Approach B (baked timeSamples) + RT2 + 50K+ prims
FIX: Switch to approach C (per-frame transform)

FAILURE: Robot floats above ground or falls through floor

CAUSE: Wrong Z offset
FIX: See navigation-primitives Robot Footprint table

FAILURE: Physics never steps after kill+relaunch

CAUSE: Kill+relaunch bug, dirty GPU state
FIX: Wait 30+ seconds, verify zero Kit processes, or reboot

FAILURE: A* path looks fine on omap but robot clips a rack at render

CAUSE: Skipped step 5/6 of the validation pipeline (see navigation-primitives)
FIX: Always validate every smoothed point is in navigable space

Integration Points

RECEIVES from: navigation-primitives — omap, A*, kinematics, robot specs, camera math
RECEIVES from: occupancy-map — map.yaml or runtime grid
RECEIVES from: urdf-mjcf-to-usd-conversion — robot USD
PRODUCES for: isaac-sim-rendering — animated scene ready for RT2 capture and frame sequences of nav runs