xb-world

xb-world — world understanding (planes, meshes, objects)

xb.world exposes the real environment to your app. See templates/8_objects, samples/depthmesh, and src/world/.

Plane detection

const options = new xb.Options();
options.enablePlaneDetection(); // detected planes become part of the world
xb.init(options);

Auto-place a model on a detected horizontal surface (e.g. the floor/a table):

xb.world.placeOnHorizontalSurface(model, /*timeout*/ {seconds: 30});

Object detection (Gemini / MediaPipe)

Object detection needs AI, the environment camera, and depth (to lift 2D detections into 3D):

options.enableAI(); // detection backend (see xb-ai — needs a key)
options.enableCamera('environment'); // video feed for the model
options.enableDepth(); // project detections into 3D
options.world.enableObjectDetection();
options.world.objects.showDebugVisualizations = true;

Run detection on demand (e.g. on pinch/click) — it resolves to an array of THREE.Object3D placed at the detected world positions:

async onSelectEnd() {
  const detected = await xb.world.objects.runDetection();
  // each entry is a THREE.Object3D with the detected world pose + metadata
  console.log('detected', detected.length);
}

Scene meshes

options.world.enableMeshDetection();
options.world.meshes.showDebugVisualizations = true; // visualize reconstructed geometry

Notes

• Backends live in src/world/objects/backends/ (GeminiDetectorBackend, MediaPipeDetectorBackend); choose via options.world.objects.
• Plane / mesh / object detection are independent — enable only what you need.
• Object detection with the Gemini backend sends camera frames to Gemini (see xb-ai and the API-key guidance in AGENTS.md).