What is StoryWeaver
StoryWeaver is an AI model launched by the Key Laboratory of Multimedia Trusted Perception and Efficient Computing of the Ministry of Education at Xiamen University and the NetEase Fuxi AI Laboratory. It enables high-quality story visualization through knowledge-enhanced story character customization. StoryWeaver uses a novel knowledge graph, Character Graph, to richly represent characters, attributes, and relationships in a story. With Customization via Character Graph (CCG) and Knowledge-Enhanced Spatial Guidance (KE-SG) technologies, it precisely injects character semantics and generates image sequences that match the textual narrative. The system performs excellently in maintaining character identity and aligning with textual semantics, effectively improving the accuracy and vividness of story visualization.
Main Features of StoryWeaver
- Character Customization and Visualization: Generates a series of visualized story images based on textual narratives and character images, precisely customizing the appearance of given characters.
- Semantic Alignment: The system can ensure that the generated image sequence is semantically consistent with the text prompts, meaning the image content matches the text description.
- Knowledge Graph Application: The system can understand and represent detailed attributes of characters in a story and the relationships between characters.
- Multi-Character Interaction: StoryWeaver can handle multi-character story scenes, keeping each character's identity clear and showing natural interactions between characters.
- Cross-Attention Allocation: Optimizes attention allocation in multi-character stories to avoid identity mixing issues.
Technical Principles of StoryWeaver
- Character Graph (CG): Constructs a knowledge graph CG composed of object nodes (characters), attribute nodes (attributes related to characters), and events (relationships between characters), which together define the core elements of a story scene.
- Customization via Character Graph (CCG): Based on CCG, transforms the structured knowledge in CG into enhanced scene descriptions, improving the consistency of character identity preservation and event semantic alignment.
- Knowledge-Enhanced Spatial Guidance (KE-SG): Introduces a knowledge encoder to extract features of different characters, adjusts the initial position priors according to character features, and modifies incorrect cross-attention maps to ensure that character knowledge accurately focuses on the corresponding regions in the story scene.
- Attention Mechanism Modification: Based on modified attention maps, enhances character-related regions and reduces attention to character-irrelevant regions, improving the visual quality of multi-character stories.
- Unified Framework: StoryWeaver provides a unified framework that can handle both single-character and multi-character story visualization tasks.
StoryWeaver Project Address
- GitHub Repository：https://github.com/Aria-Zhangjl/StoryWeaver
- arXiv Technical Paper：https://arxiv.org/pdf/2412.07375
Application Scenarios of StoryWeaver
- Educational Assistance: Uses StoryWeaver to generate story images, assisting children in learning language and literature and increasing their interest in reading.
- Comic Creation: Automatically generates framework images for comic stories, speeding up the creation process for comic artists.
- Interactive Games: In role-playing games, dynamically generates story plots and visual content based on player choices.
- Personalized Advertising: Customizes personalized story advertisements for different products, enhancing the appeal and effectiveness of advertising.
- Movie Previews: Transforms movie scripts into visual scenes, helping directors with pre-production planning and previews.