What is AnimeGamer
AnimeGamer is an infinite anime life simulation system jointly launched by Tencent PCG and City University of Hong Kong. Based on a multimodal large language model (MLLM), it supports players in immersing themselves in a dynamic game world as anime characters based on open-ended language instructions. Players can control characters such as Sosuke from Ponyo on the Cliff by the Sea and interact with the game world. The game supports generating dynamically animated shots (videos) with contextual consistency and updates to character states (such as stamina, social, and entertainment values). Compared with traditional methods, AnimeGamer performs excellently in character consistency, semantic consistency, and motion control, bringing players an immersive anime gaming experience.
Main Features of AnimeGame
- Role-playing and interaction: Players take on the role of anime characters, such as Sosuke from Ponyo on the Cliff by the Sea, and interact with the game world, allowing characters from different anime to meet and interact.
- Dynamic animation generation: Based on player instructions, dynamically animated shots (videos) are generated in real time, showing character actions and scene changes, and the animations have contextual consistency and dynamism.
- Character state updates: Based on character behavior and interactions, the character's stamina, social, and entertainment values are dynamically updated, reflecting changes in the character's state in the game world.
- Multi-turn dialogue interaction: Supports multi-turn dialogue based on natural language, and the model generates consistent game states based on historical context, providing a coherent gaming experience.
- Custom game content: Supports players in customizing their favorite characters and scenes.
Technical Principles of AnimeGame
- Multimodal large language model: AnimeGamer is based on a multimodal large language model (MLLM), understanding and generating multimodal data containing text and visual information.
- Motion-aware multimodal representation: The game decomposes animated shots into three parts: visual reference, motion description, and motion intensity. Based on an encoder, the information is integrated into a multimodal representation. Based on a video diffusion model, it is decoded into high-quality dynamic video, ensuring that the generated animated shots have contextual consistency and dynamism.
- Video diffusion model: A video diffusion model (such as CogVideoX) serves as the decoder for animated shots, decoding the multimodal representation into dynamic video. Motion intensity is introduced as an additional condition to control the amplitude of motion in the generated video, making the animation more natural and realistic.
- Contextual consistency: By taking the multimodal representation of historical animated shots as contextual input, the model can predict subsequent game states, ensuring that the generated animated shots remain consistent in context, which is crucial for maintaining the coherence and immersion of the game.
- Character state management: The game predicts updates to the character's stamina, social, and entertainment values based on the MLLM, and the state updates reflect the character's behavior and interactions in the game world. The dynamic updating of character states increases the realism and interactivity of the game.
AnimeGame Project Address
- Project official website：https://howe125.github.io/AnimeGamer.github.io/
- GitHub repository：https://github.com/TencentARC/AnimeGamer
- HuggingFace model library：https://huggingface.co/TencentARC/AnimeGamer
- arXiv technical paper：https://arxiv.org/pdf/2504.01014
Application Scenarios of AnimeGame
- Personalized entertainment: Players can choose their favorite anime characters and scenes and experience exclusive adventure stories based on language instructions.
- Creative inspiration: Provide creators with inspiration, generating character interactions and new plotlines.
- Educational assistance: Help students learn language expression and logical thinking.
- Social interaction: Players create and share anime adventure stories with friends.
- Game development: Help developers quickly generate game content and reduce development costs.