What is FairyGen
FairyGen is an animated story launched by Guangdong University of Technologyvideo generationframework, supporting the generation of animated story videos with coherent narratives and consistent style starting from a single hand-drawn character sketch. The framework uses a multimodal large language model (MLLM) for story planning, applies the character's visual style to the background based on a style propagation adapter, reconstructs the character with a 3D Agent to generate realistic motion sequences, and optimizes the coherence and naturalness of video animation based on a two-stage motion adapter. FairyGen performs excellently in style consistency, narrative coherence, and motion quality, opening up new paths for personalized animation creation.
Main features of FairyGen
- Animated story video generation: Starting from a single hand-drawn character sketch, generate animated story videos with coherent narratives and consistent style.
- Style consistency preservation: Based on the style propagation adapter, apply the visual style of the hand-drawn character to the background, ensuring style consistency throughout the entire video.
- Complex motion generation: Reconstruct the character with a 3D proxy, generate physically reasonable motion sequences, and support complex and natural motion performance.
- Narrative coherence support: With the help of a multimodal large language model (MLLM) for story planning, generate a structured storyboard, ensuring the narrative coherence of the video.
Technical principles of FairyGen
- Story planning: Use a multimodal large language model (MLLM) to generate a structured storyboard from a single hand-drawn character sketch, define a global narrative overview and a detailed shot-level storyboard, and provide the foundation for the video's narrative framework.
- Style propagation: Based on the style propagation adapter, learn the visual style of the hand-drawn character, apply it to the background, ensure visual style consistency between the background and the character, and enhance the overall visual coherence of the video.
- 3D motion modeling: Use a 3D Agent to reconstruct the character, and based on skeletal rigging and motion retargeting technology, generate physically reasonable motion sequences, providing natural and smooth motion performance for the animation.
- Two-stage motion adapter: On the basis of an image-to-video diffusion model, based on a two-stage training strategy. The first stage learns the character's spatial features and removes temporal bias; the second stage learns motion dynamics based on a temporal step-shifting strategy, ensuring the coherence and naturalness of the motion.
- Video generation and optimization: Combine the above parts and, based on a fine-tuned image-to-video diffusion model, directly render diverse and coherent video scenes aligned with the storyboard, ultimately generating a complete animated story video.
FairyGen project address
- Project official website：https://jayleejia.github.io/FairyGen/
- GitHub repository：https://github.com/GVCLab/FairyGen
- arXiv technical paper：https://arxiv.org/pdf/2506.21272
Application scenarios of FairyGen
- Education field: Teachers can use FairyGen to transform students' hand-drawn characters into animated stories, stimulating students' writing and creativity.
- Digital art creation: Artists use FairyGen to quickly generate animated videos from hand-drawn sketches, efficiently realize ideas, and save time and cost.
- Mental health treatment: Therapists use FairyGen to help patients turn hand-drawn characters into animated stories, promoting emotional expression and psychological recovery.
- Children's creative enlightenment: Parents and children use FairyGen together to turn children's hand-drawn characters into animations, cultivating imagination and creativity and enhancing parent-child interaction.
- Advertising and Marketing: Advertisers use FairyGen to create personalized animated ads, attracting audiences and improving brand communication effectiveness.