What is Wanxiang 2.1
Wanxiang 2.1 is an upgraded version launched by Alibaba'sTongyi WanxiangBased on the self-developed efficient VAE and DiT architecture, it enhances spatiotemporal context modeling capabilities, supports efficient encoding and decoding of infinite-length 1080P videos, and realizes Chinese text video generation for the first time. The upgraded Tongyi Wanxiang ranks first on the VBench leaderboard. Wanxiang 2.1 can stably display complex human movements, realistically restore real-world physical laws, generate Chinese and English video effects with one click, and possesses powerful cinematic quality and artistic style transfer capabilities. Wanxiang 2.1 supports text-to-image set generation, adopts the IC-LoRA image generation training method, enhances text-to-image context capabilities, and easily achieves combined generation of related images.
Main features of Wanxiang 2.1
- Video generation
- Complex action display: Stably displays various complex human limb movements, such as spinning, jumping, turning, rolling, etc., as well as camera movement, making video content more vivid and realistic.
- Physical law restoration: Realistically restores the physical laws of the real world, such as collisions, rebounds, cutting, squeezing, etc. For example, generating a scene where raindrops fall on an umbrella and splash water, making the video more realistic.
- Chinese and English video effect generation: Provides a variety of video effect options, such as transitions, particle effects, simulations, etc., and can generate Chinese and English video effects with one click, enhancing the visual expressiveness of videos.
- Artistic style transfer: Possesses powerful artistic style expressiveness and can transfer the cinematic quality and artistic style of videos with one click, such as film tones, impressionist brushstrokes, abstract expression, etc., generating videos in various styles.
- Image generation
- Storyboard effect restoration: Restores cinematic-level storyboard effects, ensuring consistency of elements such as characters, appearances, actions, environments, lighting, etc., and perfectly combining storyline, visual effects, and atmosphere creation.
- Four-panel comic creation: Generates four-panel comics based on the comic plot and style described by the user, quickly telling a short story.
- Creative avatar customization: Customizes exclusive couple avatars, personal avatars, etc., according to the user's preferences, style, and characteristics. Provides multiple style options such as cartoon style, realistic style, and creative style to meet users' different needs.
Technical principles of Wanxiang 2.1
- VAE architecture: Variational autoencoder (VAE) is a generative model that uses an encoder to map input data to a latent space, then uses a decoder to map the representation in the latent space back to the data space, achieving data generation and reconstruction.
- DiT architecture: The DiT (Diffusion in Time) architecture is a generative model based on diffusion models, gradually introducing noise in the time dimension and gradually removing noise to generate data. DiT can effectively capture the spatiotemporal structure of videos, support efficient encoding and decoding, and generate high-quality videos.
- IC-LoRA: IC-LoRA is an image generation training method that, based on combining image content and text descriptions, enhances text-to-image context capabilities, making the generated images more consistent with the user's text descriptions and expectations.
- Context modeling: Based on enhanced spatiotemporal context modeling capabilities, it better understands and generates video content with coherence and consistency, making elements such as actions, scenes, and styles in videos more natural and coordinated.
Project address of Wanxiang 2.1
- Project official website: Computer accessTongyi Wanxiang AI Videoofficial website, start video creation.
- Open-source address：Wan2.1
Wanxiang 2.1 video cases
Text effects
- Prompt 1: With red New Year rice paper as the background, a drop of ink appears, and the ink slowly spreads and blends. The strokes of the text have blurred and natural edges. As the blending proceeds, the ink forms the character "Fu" on the paper, with the ink color transitioning from dark to light, presenting a unique oriental charm. The background is premium and minimalist, with a magazine photography feel.
- Prompt 2: Long shot, by the Seine, brilliant fireworks bloom in the air. When the fireworks form the pink number "2025", the camera zooms in for a close-up, then they gradually dissipate.
Motion
- Prompt 1: A car speeds along a snow-covered road. The camera shoots from an aerial top-down view, showing the trees on both sides of the road and the distant mountains. The car drives on a straight road, and the surrounding snowy ground and trees form a sharp contrast. The entire scene appears especially clear under natural light, with a documentary-style realistic feel.
- Prompt 2: On the tip of an enormous ocean wave, a surfer jumps off the crest and completes an aerial spin. The camera passes through from inside the wave and emerges, capturing the moment sunlight shines through the seawater. Water splashes form a perfect arc in the air, and the surfboard leaves a trail as it cuts across the water surface. Finally, it freezes on the perfect moment of the surfer passing through the water curtain.
Special effects
- Prompt 1: A giant dragon circles above a burning city. The afterglow of the flames reflects on its scales. The camera looks up from below, capturing the dragon's figure faintly visible in the flames and smoke. Its wings beat in the air, stirring up strong winds that blow the flames in all directions.
Realistic
- Prompt 1: A close-up shot focuses on a pair of calloused hands carefully carving wood. The hands hold a carving knife, the blade glinting amid flying wood shavings. The wood grain is clear, and the fingertips move lightly between the blade, outlining delicate patterns and designs. The background is blurred, showing only the workbench and scattered woodworking tools, emphasizing the exquisite movements of the hands and the essence of craftsmanship.
- Prompt 2: Fixed 45-degree top-down shot, medium shot, shallow depth of field, morning light filtering through plane trees in dappled patches. In the center of the frame, an orange cat wearing an apron steadily raises a knife to cut cucumber, its tail swaying gently with the rhythm of chopping. In the background, cooking smoke curls upward from the stove, and sunlight slants onto the wooden cutting board and old iron wok, giving off a warm glow.
Character close-up
- Prompt 1: A close-up shot of a woman: at first she is smiling, then she becomes sad, then she begins to cry, and finally she covers her face with both hands.
Multi-person scenes/large scenes/multi-shot
- Prompt 1: At dawn, medieval style, two great armies are in melee.
Application scenarios of Wanxiang 2.1
- Film and television production: Generate ancient warfare special effects for costume dramas, such as thousands of troops charging and arrows raining down, to enhance the visual effects.
- Advertising video production: Create an advertisement for a sports drink, generating a scene of young people drinking heartily after exercise, highlighting the product's function.
- Teaching assistance: The classroom generates a video of the Battle of Red Cliffs in history class to help students understand historical events.
- Cultural creation: Produce a calligraphy art documentary, generate a video of a calligrapher's creative process, and display calligraphy techniques.
- News reporting: Report a traffic accident, generate a simulated video of the accident scene, and clearly show the course of the accident.