isaac-camera

Camera (Isaac Sim 6 / Kit 110)

Purpose

Create and configure USD cameras, render products, intrinsics, AOVs, and lens distortion models for Isaac Sim 6 sensor pipelines.

Prerequisites

Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
NVIDIA GPU with a current driver (nvidia-smi).
Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Error / symptom	Cause	Solution
Extension or import not found	Wrong `$ISAAC_SIM_DIR` or stale build	Point env vars at `_build/linux-x86_64/release` or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in `isaac-sim-troubleshooting`

Two layers stack:

UsdGeomCamera — the USD camera prim (focal length, apertures, clipping, transform).
isaacsim.sensors.experimental.rtx — the runtime sensor wrapper (authoring class + runtime sensor class). Use this for any new code that captures frames, attaches annotators, or applies lens distortion.

The legacy isaacsim.sensors.camera.Camera class still works but the extension itself is a stub (no Python). The implementation moved to isaacsim.sensors.experimental.rtx; new code should use that.

Migration: see the Migration from isaacsim.sensors.camera.Camera table below, plus the official camera migration guide for the full API change set.

When to use

Create or configure camera prims in USD stages.
Attach RGB / depth / segmentation / bbox annotators for capture.
Apply lens distortion (OpenCV pinhole, OpenCV fisheye, LUT).
Convert real-world calibration (OpenCV / ROS) to Omniverse units.
Build stereo, multi-sensor, or tiled multi-camera rigs.
Animate cameras (keyframes, motion paths, orbit, fly-through).
Generate synthetic data with Replicator randomization.
Set up Metropolis cameras for MEGA scenarios.

Fundamentals

Unit system

OpenUSD expresses optical properties in tenths of a scene unit.

Property	Units	Notes
Focal length	tenths of scene unit	`35` = 35 mm when scene units are cm
Horizontal aperture	tenths of scene unit	sensor / film width
Vertical aperture	tenths of scene unit	sensor / film height
Focus distance	scene units	perfectly-sharp distance
Clipping range	scene units	near / far clip planes

Meter stage: a 25 mm focal length is 0.025.

Coordinate system

Camera looks down -Z; +Y is up, +X is right. Same regardless of stage up-axis.

Creating and capturing — RtxCamera + CameraSensor

RtxCamera + CameraSensor is the way to create a camera and capture from it in Isaac Sim 6. RtxCamera wraps a UsdGeom.Camera prim with OmniSensorAPI applied; CameraSensor builds a Replicator render product on top and exposes annotators. Both extend the isaacsim.core.experimental.prims.XformPrim family, so the standard pose / transform methods apply.

from isaacsim.sensors.experimental.rtx import RtxCamera, CameraSensor
import isaacsim.core.experimental.utils.app as app_utils

cam = RtxCamera(
    "/World/cam",
    tick_rate=30.0,                       # 0 = autotrigger
    aux_output_level="NONE",              # RtxCamera supports NONE only
)
cam.camera.set_focal_lengths(24.0)
cam.camera.set_clipping_ranges(0.01, 1000.0)

sensor = CameraSensor(
    cam,
    resolution=(480, 640),                # (height, width)
    annotators=["rgb", "distance_to_image_plane"],
)
app_utils.play(commit=True)
data = sensor.get_data("rgb")

Author the pose directly on RtxCamera (it's an XformPrim): pass positions= / translations= / orientations= (quaternion wxyz) to the constructor, or call set_world_poses(...) afterwards.

cam.set_world_poses(positions=[[2.0, 2.0, 2.0]],
                    orientations=[[1.0, 0.0, 0.0, 0.0]])   # wxyz

TiledCameraSensor batches many cameras into one render call; useful for multi-view RL data collection. SingleViewDepthCameraSensor simulates stereo depth via OmniSensorDepthSensorSingleViewAPI.

Lens distortion (OpenCV fisheye / pinhole)

Set via API schemas + USD attributes at authoring time. The schema and attribute names are stable; use them through RtxCamera's schemas and attributes constructor args, or Apply them on the camera prim directly.

from isaacsim.sensors.experimental.rtx import RtxCamera
from pxr import Gf

cam = RtxCamera(
    "/World/fisheye",
    schemas=["OmniLensDistortionOpenCvFisheyeAPI"],
    attributes={
        "omni:lensdistortion:opencvFisheye:fx": 500.0,
        "omni:lensdistortion:opencvFisheye:fy": 500.0,
        "omni:lensdistortion:opencvFisheye:cx": 640.0,
        "omni:lensdistortion:opencvFisheye:cy": 360.0,
        "omni:lensdistortion:opencvFisheye:k1": 0.05,
        "omni:lensdistortion:opencvFisheye:imageSize": Gf.Vec2i(1280, 720),
    },
)

Equivalent OpenCV-pinhole schema: OmniLensDistortionOpenCvPinholeAPI (attributes omni:lensdistortion:opencvPinhole:{fx,fy,cx,cy,k1,k2,p1,p2,...}). LUT-based distortion uses OmniLensDistortionLutAPI.

Animation

from pxr import Usd, Gf

for frame, pos in enumerate(camera_positions):
    cam.GetPrim().GetAttribute("xformOp:translate").Set(
        Gf.Vec3d(*pos), Usd.TimeCode(frame)
    )

Replicator randomization

import omni.replicator.core as rep

with rep.new_layer():
    cameras = rep.get.prims(path_pattern="/World/Camera.*")
    with rep.trigger.on_frame(num_frames=200):
        with cameras:
            rep.modify.pose(
                position=rep.distribution.uniform((-5, -5, 2), (5, 5, 8)),
                look_at="/World/Target",
            )

Sensor checker / supported configs

For asset-driven lidar / camera configs and validation helpers, see isaacsim.sensors.experimental.rtx.SUPPORTED_LIDAR_CONFIGS and the sensor_checker module bundled with the extension. The Replicator AOV annotator names match the standard registry: rgb, distance_to_image_plane, distance_to_camera, normals, semantic_segmentation, instance_segmentation, bounding_box_2d_tight, bounding_box_2d_loose, bounding_box_3d, camera_params, pointcloud, occlusion.

Migration from `isaacsim.sensors.camera.Camera`

Full guide: isaacsim.sensors.camera → isaacsim.sensors.experimental.rtx.

Old	New
`isaacsim.sensors.camera.Camera("/World/cam", resolution=(1280, 720))`	`RtxCamera("/World/cam")` + `CameraSensor(cam, resolution=(720, 1280), annotators=["rgb"])`
`camera.set_focal_length(24)`	`cam.camera.set_focal_lengths(24.0)`
`camera.set_clipping_range(0.01, 1000)`	`cam.camera.set_clipping_ranges(0.01, 1000)`
`camera.initialize()` then `camera.get_current_frame()`	`app_utils.play(commit=True)` then `sensor.get_data(annotator)`
Carb-settings distortion (`/<path>/distortionModel`)	`OmniLensDistortion*API` schemas on the camera prim