qwen-mm-plugins-video-spatio

Video-Spatio — spatial reasoning

Use this Skill when a question needs spatial reasoning: distance, size, facing, left/right/front/behind, camera versus object motion, cross-frame counting, or viewpoint-dependent visibility. Answer appearance questions directly from the images. Read the qwen-mm-plugins-video-spatio tool schemas for arguments.

The host model supplies object boxes, distance estimates, and camera-motion estimates. Geometry tools calculate from those inputs; they do not independently measure the scene. No GPU perception server is required. VLM-backed tools use the configured OpenAI-compatible endpoint and may make several calls.

Build a scene

1. Inspect the image or sample video frames. If installed, core provides read_video and save_view. Otherwise use available image/video tools and provide local frame paths.
2. Ground the relevant objects in each frame, with one list per frame: {"label":"chair","bbox":[400,300,600,800],"depth_m":2.3}.
3. Call build_scene(frames=..., objects_by_frame=...). All frames must have the same dimensions. Use unique frame_indices when retaining original video indices.
4. For a moving camera, supply one camera_motions entry per adjacent pair. Translations are in the preceding camera's frame: forward_m is positive forward, right_m positive right, and yaw_deg positive for a right turn. Omitting motions assumes a static camera; it does not estimate motion.
5. Pass the returned scene JSON to subsequent tools, or save it and pass scene_file.

Coordinates:

• Boxes are [x1,y1,x2,y2], top-left to bottom-right, with x horizontal and y vertical. The default is 0–1000 normalized at every image resolution. Explicitly set bbox_format="pixels" or bbox_format="normalized" for pixels or 0–1 coordinates. Do not mix units in one scene.
• depth_m is a positive camera-to-object distance estimate in meters. Preserve uncertainty in the answer; visual depth estimates are not calibrated measurements.
• World coordinates use +Y up, +X right, and -Z forward from the first camera. BEV uses (x,z). pos_bev=[x,z] and yaw_deg describe each camera in that common world frame.
• The scene assumes a 60° horizontal field of view and planar camera motion. Pitch is recorded but does not change the planar geometry. Large pitch or uncertain motion makes the BEV less reliable.

Choose the analysis

Question	Tools and checks
Relative layout	visualize_bev(scene, viewpoint={"frame":N}) shows the selected camera's forward/right axes.
Virtual viewpoint	Use viewpoint={"at":"door","facing":"table"} or facing_away for the opposite direction. Resolve an object's actual facing before using its perspective.
Distance	triangulate(scene,target,frame_a,frame_b) needs the same stationary object and a real camera baseline. Check reliable; parallel, opposed, or backward rays cannot establish a reliable position.
Known-size calibration	calibrate_scale(scene,target,known_size_m,dim) returns a scale factor to apply to distance estimates.
Object motion	object_world_motion compares estimated world positions after accounting for supplied camera poses. Recheck correspondence and camera estimates before declaring movement.
Camera motion	camera_motion summarizes the poses already in the scene. It is not independent evidence validating the original motion estimates.
Facing	orient_facing(image,target) uses VLM calls. Inspect the full image for body/object orientation.
Count and identity	count_objects counts grounded instances; match_entities groups sightings across views. Inspect repeated nearby objects before accepting deduplication.
Choose frames	select_keyframes supports uniform, motion, coverage, and covisibility strategies.
Temporal changes	scene_map provides cognitive maps, appearance order, frame differences, and VLM event localization.
Visibility	view_reason reasons from an object's viewpoint. Sparse boxes cannot establish a complete occlusion model.
Motion tracks	motion segments frame differences or extrapolates supplied points. Extrapolation assumes the fitted motion continues.
Additional views	render_scene_views renders a coarse layout from the scene.

For visualize_bev, use labels that identify the intended instance unambiguously. Prefer one frame when it contains all relevant objects. A multi-frame layout additionally depends on camera estimates and correct cross-frame correspondence.

Exploration and approach suggestions

Use assess_coverage → plan_exploration when the target has not been observed, and assess_reachable when it is grounded. mobile_manip can suggest navigation, movement, staging, tracking, or search steps. These tools return suggestions; they do not move hardware.

A distance-based reachability result does not establish an obstacle-free route or a feasible grasp. For an interactive task, inspect the next observation and rebuild the scene after each movement. For an offline video, report what additional frames would resolve the question. Room priors and unobserved-space suggestions are hypotheses, not evidence that an object exists there.

Cross-check and answer

Check boxes visually before relying on geometry. verify_grounding(image,bbox,label) offers a VLM cross-check using 0–1000 normalized coordinates. Use neutral questions and allow "not visible".

Distinguish image coordinates, a particular camera's perspective, world coordinates, and an object's perspective. For a virtual viewpoint at A facing B in world (x,z):

forward = normalize(B - A)
right = [-forward_z, forward_x]
dot(C - A, forward): positive = front, negative = behind
dot(C - A, right): positive = right, negative = left

Geometry and a visualization generated from the same estimates are not independent observations. If evidence disagrees, recheck grounding, units, object identity, and camera motion. State assumptions, units, uncertainty, and limits in the answer; use the response format the user requested. Do not force a precise number or a choice when the available observations cannot support it.