SpatialClaw - A Training-Free Spatial Reasoning Framework Jointly Launched by NVIDIA and KAIST

What is SpatialClaw

SpatialClaw is a training-free spatial reasoning framework jointly launched by NVIDIA Research and KAIST. Through a code-as-action mechanism, it iteratively executes complex 3D/4D spatial reasoning tasks in a persistent Python kernel, achieving an average accuracy of 59.9% across 20 benchmarks, surpassing the previous best method by 11.2 percentage points.

Main Features of SpatialClaw

• Code-as-Action Interface: The agent writes Python code cell by cell in a persistent Jupyter kernel, freely composing outputs from perception tools such as depth reconstruction and segmentation masks.
• Five-Stage Agent Loop: The system sequentially performs planning, code generation, AST safety validation, execution and feedback assembly, and answer submission or iterative correction.
• Multimodal Perception Integration: Natively supports visual tools such as Depth Anything 3 and SAM 3, working with NumPy/SciPy for real-time geometric computation.
• Fully Training-Free Deployment: The same system prompt and toolset run directly on 6 backbone networks including Qwen3.5/3.6 and Gemma4.
• Dynamic 4D Reasoning Support: Covers spatial relationship understanding and metric computation for single images, multi-view, video, and 4D temporal scenarios.

Technical Principles of SpatialClaw

• Code-as-Action Interface: SpatialClaw replaces traditional structured tool calls with Python code generation. The VLM freely writes, executes, and corrects code in a persistent Jupyter kernel. The action space is no longer limited to predefined JSON formats and can express arbitrarily complex spatial computation logic.
• Five-Stage Agent Loop: The system sequentially performs planning, code generation, AST safety validation, execution and feedback assembly, and answer submission or iterative correction, forming a closed loop; the variable states and visualization results after execution are returned to the VLM as multimodal feedback, driving the next round of code correction.
• Persistent Jupyter Kernel: After code execution, variables and objects remain in memory, and subsequent cells can directly reuse intermediate results, supporting multi-step iterative reasoning; matplotlib output in the kernel is captured as image feedback, enabling the VLM to "see" the computation process and adjust accordingly.
• Perception Tool Integration: Visual foundation models such as Depth Anything 3 and SAM 3 are exposed as Python modules. The VLM freely orchestrates depth estimation, segmentation mask generation, and NumPy/SciPy geometric computation through code, enabling modular composition.
• AST Safety Validation: Dangerous calls are intercepted through static analysis of the abstract syntax tree, allowing only whitelisted perception tools and math libraries to execute, preventing malicious code and controlling the resource usage upper limit.

Follow WeChat and reply "Open Source" to join theAI Open Source Project Exchange Group

How to Use SpatialClaw

• Environment Preparation: Clone the GitHub repository and configure the persistent Jupyter kernel and dependency tools such as Depth Anything 3 and SAM 3.
• Model Integration: Select a supported VLM such as Qwen3.5/3.6 or Gemma4 as the reasoning backbone.
• Task Input: Submit image, multi-view, or video data containing spatial reasoning questions.
• Agent Loop: The system automatically executes an iterative loop of planning, code generation, AST validation, execution, and feedback assembly.
• Result Acquisition: Extract the final spatial reasoning answer from the kernel variable states and visualization outputs.

Core Advantages of SpatialClaw

• Training-Free Plug-and-Play: No benchmark- or model-specific fine-tuning is required, lowering deployment barriers and compute costs.
• Flexibility of Code Actions: Compared with fixed JSON tool calls, Python code can express arbitrarily complex spatial computation logic.
• Significant Performance Lead: Improves by 17.6 and 15.3 percentage points on dynamic tasks such as DSI-Bench and MindCube, respectively.
• Cross-Model Generality: The same architecture achieves consistent gains across 6 different VLMs ranging from 26B to 397B parameters.
• Safe Execution Assurance: Built-in AST safety validation prevents malicious code execution and ensures kernel state stability.

Project Address of SpatialClaw

• Project Website：https://spatialclaw.github.io/
• GitHub Repository：https://github.com/NVlabs/SpatialClaw
• arXiv Technical Paper：https://spatialclaw.github.io/static/pdfs/spatialclaw.pdf

Comparison of SpatialClaw with Similar Competitors

Application Scenarios of SpatialClaw

• Robot Navigation and Grasping: Provides mobile robots and robotic arms with precise metric distance measurement of objects and spatial relationship judgment, assisting path planning and grasping pose decisions.
• AR/VR Indoor Layout: Automatically identifies the relative positions and sizes of elements such as furniture, doors, and windows, generates reasonable indoor placement plans, and overlays them on the virtual space in real time.
• Autonomous Driving Perception: Fuses multi-camera inputs for 3D scene reconstruction, understanding the spatial topology of road elements to support decision-making and planning.
• Building Safety Inspection: Analyze the spatial compliance of construction site structures based on multi-view images, detecting safety indicators such as scaffold spacing and guardrail height.
• Industrial Precision Measurement: Non-contact three-dimensional dimensional measurement and geometric tolerance inspection of components through multi-view depth estimation.