What is WonderPlay
WonderPlay is a new framework launched by Stanford University in collaboration with the University of Utah, supporting the generation of dynamic 3D scenes from a single image and user-defined actions. Based on combining physical simulation and video generation technology, it uses a physics solver to simulate coarse 3D dynamics, then drives a video generator to synthesize more realistic videos, and uses the videos to update the dynamic 3D scene, achieving a closed loop between simulation and generation. WonderPlay supports a variety of physical materials (such as rigid bodies, cloth, liquids, gases, etc.) and a variety of actions (such as gravity, wind, point forces, etc.), allowing users to interact with scenes through simple operations and generate rich and diverse dynamic effects.
Main Features of WonderPlay
- Single-Image Dynamic Scene Generation: Using a single image and user-defined action inputs, it generates dynamic 3D scenes, showing the physical consequences of actions.
- Multi-Material Support: It covers a variety of physical materials such as rigid bodies, cloth, liquids, gases, elastomers, particles, etc., meeting diverse scene needs.
- Action Response: It supports action inputs such as gravity, wind, point forces, etc., allowing users to intuitively operate and interact with scenes, generating different dynamic effects.
- Visual and Physical Realism: Combining the precision of physical simulation and the richness of video generation, it generates dynamic scenes that conform to physical laws and are visually realistic.
- Interactive Experience: Equipped with an interactive viewer, users can freely explore the generated dynamic 3D scenes, enhancing immersion.
Technical Principles of WonderPlay
- Hybrid Generative Simulator: It integrates a physics solver and a video generator, using the physics solver to simulate coarse 3D dynamics, driving the video generator to synthesize realistic videos, and using the videos to update the dynamic 3D scene, achieving a closed loop between simulation and generation.
- Spatially Varying Dual-Modal Control: In the video generation stage, motion (flow field) and appearance (RGB) dual-modal signals are used to control the video generator, dynamically adjusting the generator's responsibility according to scene regions, ensuring that the generated video is closer to the physical simulation results in both dynamics and appearance.
- 3D Scene Reconstruction: The background and objects are reconstructed separately from the input image. The background is represented by Fast Layered Gaussian Surfaces (FLAGS), and objects are constructed as "topological Gaussian surfaces" with topological connectivity. The material properties of objects are estimated, providing a foundation for subsequent simulation and generation.
Project Address of WonderPlay
- Project Website：https://kyleleey.github.io/WonderPlay/
- arXiv Technical Paper：https://arxiv.org/pdf/2505.18151
Application Scenarios of WonderPlay
- AR/VR Scene Construction: Used to create immersive virtual environments, supporting dynamic interaction between users and scenes.
- Film and Television Special Effects Production: Rapidly generate dynamic scene prototypes, assist special effects production, and enhance visual effects.
- Education and Vocational Training: Simulate physical phenomena and work environments, enhancing the practicality of teaching and training.
- Game Development: Generate dynamic scenes and interactive effects, improving the realism and fun of games.
- Advertising and Marketing: Create dynamic advertising content, provide interactive experiences, and enhance audience engagement.