What is QianYing
QianYing is an audio game generation large model launched by Giant Network, comprising the game video generation large model YingGame and the video dubbing large model YingSound. YingGame is oriented toward open-world games and was jointly launched by Giant Network AI Lab and Tsinghua University SATLab. It can achieve interactive control of diverse character actions and physics simulation features, and when combined with YingSound, it can generate complex sound effects that match the visuals. YingSound was jointly launched by Giant Network AI Lab, Northwestern Polytechnical University ASLP Lab, and Zhejiang University. It can add realistic sound effects to silent videos and possesses video semantic understanding and temporal alignment capabilities. QianYing aims to lower the barrier to game development through AI technology, promote equality in game creation, and accelerate "game + AI" innovation.
Main Features of QianYing
- Main Features of YingGame：
- Interactive Control of Character Actions: YingGame can achieve interactive control of diverse character actions, and users can customize the actions and appearance of game characters.
- Physics Simulation Features: The model possesses powerful physics simulation features, can simulate real-world physical effects, and enhances the realism of games.
- Video Generation: Combined with the self-developed video dubbing large model YingSound, YingGame can generate complex sound effects that match the game visuals, achieving the generation of audio game videos.
- Interactive Capabilities: YingGame can understand user input interactions, such as mouse and keyboard operations, and control characters to perform various actions.
- Main Features of YingSound：
- Temporal Alignment and Video Semantic Understanding: YingSound possesses powerful temporal alignment capabilities and video semantic understanding capabilities, and can accurately add sound effects to videos.
- High-Precision Sound Effect Generation: The model can generate multiple types of high-precision sound effects for various videos, including real-world scenes, animation, and game videos.
- Diverse Application Scenarios: YingSound supports generating high-fidelity sound effects from video or a combination of video and text, and possesses generalization capability for diverse application scenarios.
- Cross-Modal Alignment Effect: Based on an innovative technical architecture and reinforcement learning, YingSound reaches industry-leading levels in generation effects, temporal alignment, and video semantic understanding.
Technical Principles of QianYing
- Technical Principles of YingGame：
- Physics Simulation Features: Integrates a physics engine to simulate real-world physical rules, such as gravity and collisions, enhancing the realism and immersion of games.
- Cross-Modal Learning: Based on cross-modal learning technology, it combines visual information with sound effect information to generate sound effects that match the game visuals.
- Technical Principles of YingSound：
- Audio-Video Synchronization Technology: Based on precise temporal alignment technology, it ensures that the generated sound effects are synchronized with the video content, involving complex time series analysis.
- Deep Neural Networks: Deep neural networks are used to generate high-precision sound effects. The neural networks can learn key features extracted from video content and convert them into corresponding sound effects.
Project Address of QianYing
- Project Official Website：ga-me.com
Application Scenarios of QianYing
- Open-World Game Development: In open-world games, create characters with complex movements and physical interactions, providing a more realistic and immersive gaming experience.
- Game video content creation: Game developers generate high-quality game trailers, cinematic animations, and other video content to enhance the game's market appeal.
- Game education and training: In the field of education, create simulated environments and scenarios for use in skills training or safety education to improve learning efficiency and effectiveness.
- Game live streaming and esports: Add special effects and sound effects to game live streams to enhance their watchability and interactivity, while also providing a more vivid viewing experience for esports events.
- Game testing and prototyping: Rapidly generate game prototypes to accelerate the game design and testing process and reduce development time and costs.