WorldVLA - An Autoregressive Action World Model Jointly Launched by Alibaba DAMO Academy and Zhejiang University

What is WorldVLA

WorldVLA is an autoregressive action world model jointly launched by Alibaba DAMO Academy and Zhejiang University. The model integrates the Vision-Language-Action (VLA) model and the world model into a single framework. The model predicts future images based on actions and image understanding, aiming to learn the fundamental physical laws of the environment to improve action generation. The action model generates subsequent actions based on image observations, assists visual understanding, and in turn helps the visual generation of the world model. WorldVLA outperforms independent action models and world models in performance, highlighting the mutual enhancement between the world model and the action model. To address the performance degradation issue when generating a series of actions in an autoregressive manner, an attention masking strategy is proposed, which selectively masks previous actions when generating the current action, significantly improving performance in action chunk generation tasks.

Main Features of WorldVLA

• Action Generation: Generates subsequent actions based on images and language instructions, supporting continuous action planning.
• Image Prediction: Predicts future image states based on the current image and actions, improving visual prediction accuracy.
• Environment Understanding: Learns the physical laws of the environment, enhancing visual and action understanding capabilities.
• Bidirectional Enhancement: The action model and the world model mutually reinforce each other, improving overall performance.

Technical Principles of WorldVLA

• Unified Framework: WorldVLA integrates the Vision-Language-Action (VLA) model and the world model into a single framework. It uses three independent encoders (image encoder, text encoder, and action encoder) to encode data from different modalities into tokens in a unified vocabulary, enabling cross-modal understanding and generation.
• Autoregressive Generation: The model generates actions and images in an autoregressive manner. The action model generates actions based on historical images and language instructions, and the world model predicts future image states based on historical images and actions.
• Attention Masking Strategy: To address the potential performance degradation issue when autoregressive models generate a series of actions, WorldVLA proposes an attention masking strategy. It selectively masks previous actions when generating the current action, reducing error propagation and improving the performance of action chunk generation.
• Bidirectional Enhancement: WorldVLA achieves bidirectional enhancement based on the interaction between the world model and the action model. The world model helps the action model better understand the physical laws of the environment by predicting future states, and the action model helps the world model more accurately predict future image states by generating actions.
• Training Strategy: During training, WorldVLA mixes action model data and world model data to ensure that the model can simultaneously learn the capabilities of action generation and image prediction. The mixed training strategy allows the model to achieve multiple functions within a single architecture.

Project Addresses of WorldVLA

• GitHub Repository：https://github.com/alibaba-damo-academy/WorldVLA
• HuggingFace Model Library：https://huggingface.co/collections/jcenaa/worldvla-685b9df63bdfe8cb67cc71b2
• arXiv Technical Paper：https://arxiv.org/pdf/2506.21539

Application Scenarios of WorldVLA

• Robot Goal-Oriented Tasks: Helps robots complete goal-oriented tasks based on visual and language instructions, such as moving an object from one position to another.
• Fine Operations in Complex Environments: In complex environments, such as cluttered desks or narrow spaces, generate highly adaptable actions to complete fine operations.
• Human-robot collaboration tasks: In human-robot collaboration scenarios, understand human actions and intentions, generate corresponding collaborative actions, and improve collaboration efficiency.
• Future scenario simulation and prediction: Predict future image states to help robots plan ahead and evaluate the consequences of actions, such as road scene prediction in autonomous driving.
• Education and research platform: As a teaching tool and research platform, help students and researchers understand and practice the principles of robot control and visual prediction.