01 Tencent is ready
Tencent has finally launched its own AI video — "Hunyuan Video Model」。
Recently, I was invited to participate in the beta test of the Hunyuan Video Model. I worked on it all weekend, from morning to night, and ran a total of over 300 videos
。
First, the conclusion:As Tencent's first version (text-to-video, 5s), the overall quality is very high. Ininstruction following, motion and image stability, camera language, photorealistic texture, physical adherenceand other aspects, it performs well,very few bad generations。
Even, in someshot transitions, action effects, sci-fi/fantasy styles, abstract understandingand other aspects, there are surprising performances.
Please watch the VCR:
Experience path:Tencent Yuanbao APP - AI Applications - AI Video.
02 Hands-on testing of 10 styles, 30 Cases
In order to systematically test the quality of the Hunyuan Video Model, of course, compared with those professional evaluation benchmarks, it is not very systematic. It is just some style scenarios that I personally think are important and that everyone often uses. I divided them into 10 sections for testing.
These 10 styles are:close-up, photorealistic, people, animals, sci-fi, special effects, animation, art/abstract, sports, multi-person scenes/large scenes/multi-shot。
For each style, design 3~5 prompts respectively, let Hunyuan generate videos, and conduct the evaluation.
For the prompt part, I first come up with an idea myself, describe it in one sentence, and then let AI help me optimize and expand it. After AI optimization, I revise the prompt myself, and basically it can be sent to the model to start running.
The prompt framework generally cannot do without these templates.
- Template 1:Prompt = subject + scene + motion
- Template 2:Prompt = subject (subject description) + scene (scene description) + motion (motion description) + (camera language) + (atmosphere description) + (style expression)
- Template 3:Prompt = subject + scene + motion + (style expression) + (atmosphere description) + (camera movement method) + (lighting) + (shot size)
Focus on subject + scene + motion is enough. If you are not good at describing the other parts, you can also select them through the tags provided in the backend.
Without further ado, let's directly look at the Cases that were run.
Ps. All Cases were personally tested by me and do not include any official demos.
(1) Photorealistic
Photorealistic is almost a must-test style for video models. The main focus is on the model's generation effects for different scenes, character expressions, character actions, texture details, and light and shadow changes, and whether they are consistent with the real world.
1) A woodpecker is pecking a hole in a tree, photorealistic style.
2) A Chinese beauty wearing Hanfu, her hair fluttering in the wind. Then the camera switches to a frontal close-up. The background is Zhangjiajie.
3) A penguin wearing a red scarf walking in a sea of flowers, the red scarf forming a sharp contrast with the colors of the sea of flowers. The sea of flowers in the background sways gently in the wind, petals fall, and morning dew sparkles.
4) Ultra-telephoto lateral movement, abandoned industrial factory building, key light seeping in from a shattered skylight, natural light.
(2) Close-up
Close-ups are a style that video models are relatively good at. The key to competition among different models lies in their ability to present details, such as details of object movement, details of human body movements, details of facial expressions, details of image quality, etc.
A good close-up shot can easily bring the audience closer to the protagonist and make them feel immersed in the scene.
5) A man looks into the distance in terror, with a burning and exploding city in the background. The camera is aimed at the man's face, capturing his terrified expression.
6) The camera slowly pushes in. The background is a small and cozy living room, where a young woman sits on the sofa, completely focused on reading. A steaming teacup is placed on the coffee table.
7) A strange and terrifying ancient creature crawls through the mud.
(3) People
For people, the main focus is on the realism of the video model's portrayal of human skin tone, body movements, facial expressions, and clothing, which is also where we humans can most easily identify whether AI is real or fake.
But then again, text-to-video does not perform very well in terms of people. To make people perform more stably, realistically, and consistently, it is generally necessary to generate through image-to-video.
8) A little boy is completely focused on assembling building blocks.
9) A little girl holding a balloon slowly runs forward.
10) A man sits on the sofa watching TV, then holds his head with both hands, with a very surprised expression.
(4) Animals
Compared with people, different video models perform much better on animals. But the premise is that your animal must move "boldly," rather than just zooming the image in and out.。
From the many cases I have run, the Hunyuan video model is very good at animal realism, with a bit of a documentary flavor.
11) On the African savanna, a cheetah is running at extreme speed, chasing an antelope.
12) In the Greater Khingan Mountains, a tiger is running at extreme speed, with a snow-covered forest in the background.
13) On a branch in front of the red walls of the Forbidden City, a magpie is foraging for food.
(5) Sci-fi, magic, and fantasy
Fantasy styles such as sci-fi, magic, and fantasy are an important reason many people are attracted to using AI to make videos, of course including me.
Fantasy style especially tests the video model's dataset and generalization ability (referring to the model's ability to perform on new, unseen data), and whether it can present some fantasy scenes, such as changes in light and shadow, changes in color, transformation effects, action effects, etc.
In this part, I included the most cases. Considering that videos converted to images are compressed, for some cases I directly included the original videos.
14) A spaceship is passing through an asteroid belt.
15) A spaceship is passing through a time tunnel, surrounded by colorful light rays.
16) Two giant robots are fighting fiercely in the city, and every collision produces a huge shockwave, shattering nearby buildings into pieces.
17) In a dim corridor, a Marine Corps unit is passing through an abandoned corridor.
18) In the faintly visible clouds, dark clouds gather and lightning flashes and thunder rumbles. Suddenly, a giant dragon passes through the clouds and comes rushing over.
With this kind of imagination, Hunyuan must have "watched" Game of Thrones many times.
(6) Special effects
Special effects,Special Effects, are the most important visual art in film and television. Common special effects include explosions, smoke, fire, extreme speed, etc.
Special effects shots also mainly test the generalization ability of the video model, examining the model's degree of instruction following and its ability to render details.
19) In a snowstorm, a steam train travels through rugged mountains, black smoke billowing from the locomotive straight into the sky, the carriages leaving deep tracks in the vast white snow.
20) Inside a dilapidated warehouse, an explosion suddenly occurs.
21) On a foggy night, with bright moonlight, a medieval sailing ship sails on the sea, filled with an eerie atmosphere.
22) Colorful jellyfish swim freely on the seabed. Their bodies appear in transparent blue, purple, and pink, emitting a captivating glow in the water.
(7) Animation
Animation mainly tests the model's support for various styles and its aesthetics, such as 2D, 3D, vector, clay, ink wash, Hayao Miyazaki, Disney, and so on.
First, here is a Sora prompt.
23）Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle. The art style is 3D and realistic, with a focus on lighting and texture. The mood of the painting is one of wonder and curiosity, as the monster gazes at the flame with wide eyes and open mouth. Its pose and expression convey a sense of innocence and playfulness, as if it is exploring the world around it for the first time. The use of warm colors and dramatic lighting further enhances the cozy atmosphere of the image.
Now let's look at the Hayao Miyazaki style.
24) A fantastical garden comes into view. In the garden grow all kinds of exotic flowers and plants, varied in form and rich in color. In the garden also live a group of lively and adorable little spirits, dressed in colorful clothes, playing among the flowers and plants. Ghibli animation style, making people feel as if they are in a dreamlike world created by Hayao Miyazaki.
(8) Art/Abstract
Artistic style mainly tests the video model's abstract understanding of graphics, space, color, and changes in force. After testing several cases, I didn't expect Hunyuan could also create some abstract art videos.
25) Particles rotate and converge into abstract forms.
26) Different colors form irregular shapes, slowly rotating.
27) Fixed shot at a 5-degree angle, shallow depth of field focus, purple-red neon lights interwoven with cyan holographic projections. In the center of the frame, a mechanical dancer dressed avant-garde spreads both arms and thanks the audience.
(9) Motion
Motion is regarded as the jewel in the crown of video models because it is the most challenging.
To generate videos that conform to real-world physical motion, the model must have quite deep technology in its understanding of spatial positional relationships, its handling of changes in force and form for different objects, and its semantic understanding of different objects and different motions, in order to generate videos that follow the rules of physics.
28) On an off-road track at sunset, a modified Ford F-150 Raptor roars past. The raised suspension lets the huge puncture-proof tires churn freely on the muddy ground, and mud splashes onto the roll cage, forming mottled patterns. The body decals glitter in the golden sunlight, and the whine of the supercharger interweaves with the roar of the exhaust.
29) Slow-motion spinning shot, thunderstorm weather accompanied by lightning, a heroic and dashing Chinese swordsman dances with a sword in the rain. The background is a bamboo forest.
30) An off-road vehicle drives on a steep mountainside, and in the distance Mount Gongga slowly rises visually and gradually becomes clear.
(10) Multi-person scenes/large scenes/multiple shots
Multi-person scenes involve the coordination of multiple characters' actions as well as computing power issues. At present, basically many video models collapse on these, including Gen3, Kling, and others. Let's see how Hunyuan performs.
31) The camera slowly rises from a partial close-up of the legs of a knight riding a horse, and finally captures the knight's face; the knight looks forward with a resolute expression. The background is a medieval battlefield, where two armies are fighting, with men and horses thrown into confusion.
32) A group of people sit around a campfire, talking and laughing, full of joyful laughter.
Having tested 10 style domains, let's make a summary:
1) The Hunyuan model follows instructions (that is, prompts) relatively well.Going forward, when designing prompts, it is recommended to have strong visual logic and clear instructions, and do not pile up a bunch of modifiers and too many subject words.
Otherwise, it will instead interfere with the model's attention, that is, the T in the model's DiT architecture, Transformer, the self-attention mechanism.
2) Dynamic performance and image stability are very good.Among the more than 300 videos I tested, there were certainly failed cases, but not a single video was doing a PPT-style zoom in or zoom out. They were all normal actions at normal speed, with very few slow-motion shots or PPT animations.
3) It understands cinematic language well.If you specify what kind of shot and shot size, the model will strictly follow it. If you do not specify, the model will understand it on its own based on the prompt and design the shots, and sometimes it can be surprising.
For example, this one is really nice.
Prompt:A huge ocean wave, a surfer jumps on the crest and completes a mid-air spin. The camera passes through from inside the wave and emerges, capturing the moment sunlight shines through the seawater. Water splashes form a perfect arc in the air, and the surfboard leaves a trail as it cuts across the water's surface. Finally, it freezes on the perfect moment when the surfer passes through the curtain of water.
4) A 5s video can also cut shots.In some prompt scenarios (usually long prompts), even with only a 5s video, the Hunyuan model can automatically cut shots. After cutting shots, it can still maintain subject consistency.
5) It performs excellently in styles such as sci-fi, fantasy, realistic documentary, special effects, and sports, with a very high success rate.Especially the fantasy style, which has a strong Game of Thrones flavor. It is speculated that this should be related to Tencent's own video dataset.
6) Fewer generation attempts are needed.If the instructions are clear, sometimes a satisfactory video can be obtained in one generation. At worst, after generating 3-5 more times, you can basically also get a satisfactory video.
7) It tries to accommodate beginners.In the input box interface, it provides style, shot size, lighting, camera movement, and multiple modes (smooth camera movement, rich actions, director mode), so beginners can also get started quickly.
Do not underestimate these tags. During my testing, these tags helped my video results a lot, especially in terms of video style and camera movement.
Of course, some shortcomings were also found during testing.
1) Generalization ability needs improvement.Some unfamiliar, niche, and untrained descriptive words (such as subject, scene, action, etc.) still cannot be recognized by Hunyuan, which affects the model's creativity to a certain extent.
2) Image quality still needs improvement. Currently it is only 720P (really 720P). Although a "high quality" mode is provided, for professional creators, it is still not quite enough.
3) The understanding of local people still needs improvement.If the prompt does not specify "Asian", the model usually generates Europeans. Of course, text-to-video is inherently not good at subject consistency for people, and to improve character consistency, we still have to wait for image-to-video. In addition, the model is also slightly weaker in showing emotions.
03 Written at the end
After three consecutive days of evaluation, I personally think that,as a first-generation model, Hunyuan's overall quality is very high, better than the first version of many video models.
After learning a bit from the Hunyuan team, this comes from their innovations in these areas:
- It uses a new-generation language model as the text encoder, with stronger semantic understanding and visual presentation capabilities;
- It uses a full attention mechanism throughout, rather than spatiotemporal modules, making the transitions between each frame of the video smoother;
- It uses a self-developed hybrid image-video VAE (3D variational encoder) to improve the model's performance on details, such as faces, fingers, high-speed shots, etc.
And even more critically, Tencent announced that this model will be open-sourced!!
From now on, whether individuals or enterprises,all developers can use this model for free onHugging Faceand Github.
Magnificent, truly magnificent! A 13-billion-parameter model, open-sourced just like that. Including the model weights, inference code, model algorithms, and other complete model components, all directly made public.
You should know that because video models are the most technically difficult, there really are not many who dare to open-source them or can open-source them, including the video model Movie Gen launched by "Source God" Llama, which does not plan to be open-sourced.
The Hunyuan video model is open-sourced as soon as it launches. This bearing, this vision, awesome. So far, they have already open-sourced text-to-text, text-to-image, 3D generation, and the latest text-to-video.
Tools involved in this article
Tencent Hunyuan text-to-video:https://ai-bot.cn/video-hunyuan-tencent/
Original link:Hands-on test, Tencent Hunyuan's latest video model