What is Aether
Aether is an open-source generative world model by Shanghai AI Lab, trained entirely on synthetic data. Aether is the first to deeply integrate 3D spatiotemporal modeling with generative modeling, possessing three core capabilities: 4D dynamic reconstruction, action-conditioned video prediction, and goal-oriented visual planning. Aether can perceive the environment, understand object positions and motion relationships, and make intelligent decisions. Aether demonstrates strong zero-shot generalization capabilities in the real world, using virtual data training to efficiently complete complex tasks, providing powerful spatial reasoning and decision support for embodied intelligence systems.
Aether's Main Features
- 4D Dynamic Reconstruction: Reconstructs a 3D scene model that includes time and space from videos, capturing dynamic changes.
- Action-Conditioned Video Prediction: Predicts future scene changes based on initial observations and action trajectories.
- Goal-Oriented Visual Planning: Generates reasonable paths based on start and goal scenes, assisting intelligent systems in planning action routes.
Aether's Technical Principles
- Unified Multi-Task Framework: Integrates the three tasks of dynamic reconstruction, video prediction, and action planning into a unified framework for optimization. Based on interleaved task feature learning, it achieves collaborative optimization between different tasks, improving the model's stability and robustness.
- Geometry-Aware Modeling: Introduces 3D spatiotemporal modeling, constructs geometric space to enhance the model's spatial reasoning capabilities. Uses massive simulated RGBD data (color images and depth maps), develops a complete data cleaning and dynamic reconstruction pipeline, and annotates rich action sequences.
- Camera Trajectory as Action Representation: Selects camera trajectory as the representation of global actions. In navigation tasks, the camera trajectory directly corresponds to the navigation path; in robotic manipulation, the motion of the handheld camera can capture the 6D motion of the end effector.
- Diffusion Models and Multimodal Fusion: Based on a pre-trained video diffusion model, post-trained with synthetic 4D data. Converts depth videos into scale-invariant normalized disparity representations, and encodes camera trajectories into scale-invariant ray map sequence representations aligned with the spatiotemporal framework of Diffusion Transformers (DiTs). Based on dynamically integrating cross-task and cross-modal condition signals, Aether achieves fusion and collaborative optimization of multimodal information.
- Zero-Shot Generalization Capability: Trained entirely on virtual data, achieving zero-shot generalization to the real world. Based on combining different condition inputs (such as observation frames, goal frames, and action trajectories) with the diffusion process, it achieves unified modeling and generation for multiple tasks. It allows the model to transfer to real-world scenarios with excellent performance without real-world data.
Aether's Project Address
- Project Website：https://aether-world.github.io/
- GitHub Repository：https://github.com/OpenRobotLab/Aether
- HuggingFace Model Library：https://huggingface.co/AetherWorldModel/AetherV1
- arXiv Technical Paper：https://arxiv.org/pdf/2503.18945
- Online Demo：https://huggingface.co/spaces/AmberHeart/AetherV1
Aether's Application Scenarios
- Robot Navigation: Helps robots plan paths and avoid dynamic obstacles.
- Autonomous Driving: Real-time reconstruction of road scenes and prediction of traffic dynamics.
- Virtual Reality: Generates immersive virtual scenes to enhance user experience.
- Industrial Robots: Optimizes robot operation paths and improves production efficiency.
- Intelligent Surveillance: Analyze surveillance video to predict abnormal behavior.