mujoco-rendering

MuJoCo Rendering: Cameras, Contexts & Offscreen Buffers

[!TIP]

Related skills:

• mujoco-python — Core physics simulation, stepping, and state management.
• mujoco-studio — Interactive simulation, Studio viewer apps, and plugin architecture.
• mujoco-gui — Dear ImGui UI design and custom controls.

1. Overview: Headless vs Interactive Rendering

MuJoCo provides two distinct graphics workflows:

• Headless Offscreen Rendering (mujoco.Renderer / mjrContext): Focused on capturing image buffers (RGB, depth, object segmentation) without opening a desktop window. Ideal for batch dataset collection, reinforcement learning observations, and automated visual regression testing.
• Interactive Viewers (mujoco.experimental.studio / mujoco.viewer): Interactive desktop or web-streamed applications designed for real-time inspection, camera manipulation, and Dear ImGui control panels.

2. Headless Contexts on Linux: MUJOCO_GL

On headless Linux systems (servers, containers), configure the OpenGL context provider via the MUJOCO_GL environment variable before importing mujoco:

import os
os.environ['MUJOCO_GL'] = 'egl'      # Hardware-accelerated offscreen on NVIDIA GPUs
# os.environ['MUJOCO_GL'] = 'osmesa' # Portable CPU software rasterization

import mujoco  # Must import AFTER setting MUJOCO_GL

Or from the command line:

MUJOCO_GL=egl python3 my_script.py

3. The Python Renderer Class (RGB, Depth & Segmentation)

mujoco.Renderer manages GL context creation, scene geometry synchronization, and buffer readback. Always use it as a context manager:

import mujoco
import numpy as np

model = mujoco.MjModel.from_xml_path('scene.xml')
data = mujoco.MjData(model)

with mujoco.Renderer(model, height=480, width=640) as renderer:
  mujoco.mj_forward(model, data)

  # 1. RGB Rendering
  renderer.update_scene(data, camera='overhead_cam')
  rgb = renderer.render()  # Shape: (480, 640, 3), dtype: uint8

  # 2. Metric Depth Rendering (distance in meters)
  renderer.enable_depth_rendering()
  renderer.update_scene(data, camera='overhead_cam')
  depth = renderer.render()  # Shape: (480, 640), dtype: float32
  renderer.disable_depth_rendering()

  # 3. Semantic Segmentation Rendering
  renderer.enable_segmentation_rendering()
  renderer.update_scene(data, camera='overhead_cam')
  seg = renderer.render()  # Shape: (480, 640, 2), dtype: int32
  # seg[:, :, 0] = object ID (-1 for background)
  # seg[:, :, 1] = object type (mjtObj enum, e.g. mjOBJ_GEOM)
  renderer.disable_segmentation_rendering()

Rendering Inside Simulation Loops

Creating a Renderer allocates GPU framebuffers. In RL or evaluation loops, create the renderer once outside the loop:

with mujoco.Renderer(model, height=84, width=84) as renderer:
  for step in range(1000):
    mujoco.mj_step(model, data)
    if step % render_interval == 0:
      renderer.update_scene(data, camera='agent_cam')
      # Call .copy() because the renderer's internal buffer is reused
      frame = renderer.render().copy()

4. C++ Offscreen Rendering Pipeline (mjr & mjv)

In C++, offscreen rendering is performed using the mjvScene and mjrContext APIs:

#include <mujoco/mujoco.h>

// 1. Initialize visualization structures
mjvCamera cam;
mjv_defaultCamera(&cam);
cam.type = mjCAMERA_FIXED;
cam.fixedcamid = mj_name2id(m, mjOBJ_CAMERA, "overhead_cam");

mjvOption opt;
mjv_defaultOption(&opt);

mjvScene scn;
mjv_makeScene(m, &scn, 2000);  // Buffer up to 2000 geoms

// 2. Initialize offscreen rendering context
mjrContext con;
mjr_defaultContext(&con);
mjr_makeContext(m, &con, mjFONTSCALE_100);

// 3. Render frame
mjrRect viewport = {0, 0, 640, 480};
mjv_updateScene(m, d, &opt, nullptr, &cam, mjCAT_ALL, &scn);
mjr_render(viewport, &scn, &con);

// 4. Read pixels into user buffers
unsigned char rgb[640 * 480 * 3];
float depth[640 * 480];
mjr_readPixels(rgb, depth, viewport, &con);

// 5. Clean up
mjr_freeContext(&con);
mjv_freeScene(&scn);

5. Camera Definition & Placement

MuJoCo cameras look down their negative Z-axis:

• -Z: Viewing forward direction
• +X: Right
• +Y: Up

Defining Cameras in MjSpec

# Fixed camera pointing down at ground plane
cam = spec.worldbody.add_camera(
    name='overhead',
    pos=[0, 0, 3.0],
    quat=[0.707, 0.707, 0, 0],  # 90 deg rotation around X
    fovy=60,
)

# Camera attached to robot wrist
wrist_cam = wrist_body.add_camera(
    name='wrist_cam',
    pos=[0.05, 0, 0],
    xyaxes=[0, -1, 0, 0, 0, -1],
    fovy=75,
)

Free Camera (MjvCamera)

For custom interactive or orbital viewpoints:

cam = mujoco.MjvCamera()
cam.type = mujoco.mjtCamera.mjCAMERA_FREE
cam.lookat[:] = [0, 0, 0.5]
cam.distance = 2.5
cam.azimuth = 90.0
cam.elevation = -20.0

renderer.update_scene(data, camera=cam)

6. Framebuffers & Scene Options

Offscreen Framebuffer Resolution

The offscreen rendering resolution cannot exceed the dimensions pre-allocated in the model. Adjust the buffer limit in the spec before compilation:

spec.visual.global_.offwidth = 1920
spec.visual.global_.offheight = 1080
model = spec.compile()

The Single <visual> Block Rule

MuJoCo uses only the last <visual> block defined in an XML file. Multiple visual blocks will silently overwrite earlier settings:

<!-- ❌ WRONG — second visual block silently resets offwidth/offheight to default -->
<visual><global offwidth="1920" offheight="1080"/></visual>
<visual><headlight ambient="0.3 0.3 0.3"/></visual>

<!-- ✅ RIGHT — consolidate into one visual block -->
<visual>
  <global offwidth="1920" offheight="1080"/>
  <headlight ambient="0.3 0.3 0.3"/>
</visual>

Geom Group Visibility & Visualization Flags

Control rendering layers via MjvOption:

opt = mujoco.MjvOption()
opt.geomgroup[:] = False   # Hide all groups
opt.geomgroup[0] = True    # Show primary geometry (default group)
opt.geomgroup[1] = True    # Show secondary visuals

# Toggle visual debugging flags
opt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
opt.flags[mujoco.mjtVisFlag.mjVIS_TRANSPARENT] = True

renderer.update_scene(data, camera='overhead', scene_option=opt)

7. Filament & Modern Graphics Backends

MuJoCo supports the Filament physically-based rendering (PBR) engine experimentally for advanced lighting, soft shadows, and materials. In Python, Filament renderer bindings are available under mujoco.rendering.filament.

8. Common Gotchas & Best Practices

1. GL context leaks: Always close the renderer or use a with block. Failing to free renderers in loops exhausts GL contexts.
2. Destruction order: In C++, mjr_freeContext must be called before the underlying OpenGL context is destroyed.
3. Buffer re-use: renderer.render() returns an internal buffer view. Call .copy() if storing frames in a list.
4. Camera lookup by name: Pass camera='camera_name' or the integer ID. Avoid passing raw string IDs that are not present in the model.

9. Key References

Documentation

• Visualization & Rendering Reference
• Python Bindings Reference

Source Code Examples

• C API Header