Spatial-TTT - Open-source streaming visual spatial intelligence framework by Tsinghua University and Hunyuan

What is Spatial-TTT

Spatial-TTT is a streaming visual spatial intelligence framework jointly launched by Tsinghua University, Tencent Hunyuan, and Nanyang Technological University. The framework has only 2B parameters and, through Test-Time Training (TTT) technology, continuously updates its internal spatial memory while watching a video stream, enabling spatial reasoning over videos up to 120 minutes long. On multiple benchmarks such as VSI-Bench and MindCube-Tiny, its performance exceeds GPT-5、Gemini-3-pro and other closed-source large models, with peak GPU memory usage saving over 40% compared to industry-leading solutions.

Main features of Spatial-TTT

• Streaming spatial memory maintenance: Inputs the video stream in chunks, updates model parameters online (fast weights), and continuously accumulates 3D spatial evidence without needing to load the entire video at once.
• Long-horizon spatial reasoning: Supports tasks such as relative/absolute distance estimation, object counting and size judgment, room size perception, direction recognition, path planning, and appearance order reasoning.
• Ultra-long video processing: Can stably process continuous video streams from 10 minutes to 120 minutes, maintaining performance without degradation on the VSI-SUPER long-term memory benchmark.
• Efficient inference: With 1024-frame input, peak GPU memory is only 11.9GB, and theoretical computation is 799.4 TFLOPs, significantly lower than comparable large models.

Follow WeChat and reply "open source" to join theAI open-source project exchange group

Technical principles of Spatial-TTT

• Hybrid TTT architecture: The research team interleaves TTT layers and standard self-attention anchor layers in the decoder at a 3:1 ratio. 75% of the TTT layers are responsible for writing long-range spatial information into fast weights, achieving memory expansion with linear complexity; 25% of the full-attention layers preserve the pretrained model's semantic understanding and cross-modal alignment capabilities, preventing a pure TTT structure from damaging the original vision-language abilities.
• Spatial prediction mechanism: Traditional TTT uses pointwise linear projections to generate Q
  /K/V, ignoring the local geometric structure of visual tokens. Spatial-TTT introduces lightweight 3D spatiotemporal convolutions into the TTT branch, so that fast weights learn the predictive relationship from spatiotemporal context to spatiotemporal context, rather than the mapping of isolated tokens, stably capturing geometric correspondences, viewpoint changes, and temporal continuity.
• Dense scene description supervision and progressive training: Existing spatial intelligence data is mostly sparse local question answering, which is difficult to drive fast weights to learn global updates. The research team constructed dense 3D scene description data covering global context, object categories and counts, and spatial relationships, and adopted two-stage spatial perception progressive training: the first stage learns to "remember the entire space" through dense descriptions, forming global 3D awareness; the second stage uses millions of spatial VQA data to strengthen streaming reasoning capabilities.

How to use Spatial-TTT

• Environment preparation: Visit the GitHub repository (https://github.com/THU-SI/Spatial-TTT/）克隆代码，配置 Python environment and dependencies.
• Model loading: Load the pretrained Spatial-TTT-2B weights and initialize the TTT fast weights.
• Input video stream: Split the long video into consecutive chunks and input them to the model chunk by chunk; after processing each chunk, the model automatically updates its internal spatial state.
• Spatial question answering: Input a natural-language spatial question (e.g., "starting from the red candle and facing the window, how do I get to the door"), and the model generates an answer based on accumulated spatial memory.
• Extended Applications: It can connect to real-time video streams from robots, autonomous driving, or AR devices to achieve continuous spatial perception.

Core Advantages of Spatial-TTT

• Small Parameters Surpass Large Models: 2B parameters beat closed-source giants such as GPT-5 and Gemini-3-pro on multiple spatial benchmarks, and MindCube-Tiny leads Gemini-3-pro by 12 percentage points in accuracy.
• Linear Scalability: Through online updates of fast weights, memory usage and computation grow linearly with video length, avoiding the quadratic complexity explosion of traditional long-context models.
• Spatial Memory Rather Than Content Accumulation: It does not rely on an ever-expanding context window, but instead transforms observations into an internal spatial state that can be updated, corrected, and queried, making it more suitable for long-term operation of physical agents.
• Significantly Improved Memory Efficiency: In a 1024-frame scenario, memory usage is 11.9GB, more than 40% lower than the industry-leading solution (21.2GB), and comparison models with explicit geometric encoders can no longer run at the same frame count.

Spatial-TTT Project Address

• Project Website：https://liuff19.github.io/Spatial-TTT/
• GitHub Repository：https://github.com/THU-SI/Spatial-TTT
• HuggingFace Model Library：https://huggingface.co/collections/THU-SI/spatial-ttt
• arXiv Technical Paper：https://arxiv.org/pdf/2603.12255

Comparison of Spatial-TTT with Similar Competitors

Application Scenarios of Spatial-TTT

• Robot Navigation: When service robots or home robots continuously move around a room, they update the spatial map in real time, answer "the position of the sofa relative to the window," or plan a path to a target point.
• Autonomous Driving: During long-duration driving, the vehicle continuously accumulates the spatial relationships among roads, intersections, and obstacles, supporting complex direction judgment and distance estimation.
• AR/VR Spatial Interaction: During use, head-mounted devices continuously understand the environmental layout, enabling virtual-real fused spatial anchoring and object relationship queries.
• Intelligent Monitoring and Security: It performs online spatial analysis on long video streams, tracking the order of object appearances, counts, and position changes without the need for offline full processing.