What is MM-StoryAgent
MM-StoryAgent is an open-source multimodal, multi-agent framework jointly launched by the X-LANCE Lab of Shanghai Jiao Tong University and Alibaba Group, used to generate immersive narrated storybook videos. Based on combining large language models (LLMs) and generation tools of multiple modalities (such as text, images, and audio), it uses a multi-stage writing process and modality-specific prompt revision mechanisms to enhance the appeal and immersion of stories. The framework supports flexible modular design and can replace different generation models and APIs. MM-StoryAgent improves story quality and achieves better alignment among images, speech, music, and sound effects, providing an efficient, flexible, and expressive solution for the automated creation of children's storybooks.
Main features of MM-StoryAgent
- High-quality story generation: Based on multi-agent collaboration and a multi-stage writing process, it generates engaging, educational, and emotionally resonant story content.
- Multimodal content generation: It combines multiple modalities such as text, images, speech, music, and sound effects to bring users a rich immersive experience.
- Character consistency: In image generation, based on character extraction and prompt revision, it ensures that characters in the story remain visually consistent.
- Modality alignment: Based on prompt revision and contrastive learning models, it optimizes the alignment between text and images and audio, improving the coherence of the overall story.
- Flexible modular design: It supports flexible replacement of generation modules (such as different text generation models, image generation models, etc.), making it easy for developers to customize and optimize according to their needs.
Technical principles of MM-StoryAgent
- Multi-agent collaboration architecture: It simulates dialogue between amateur authors and experts to generate story outlines and chapter content. The dialogue process simulates human brainstorming, providing richer creativity and appeal for the story. For the needs of different modalities such as images and audio, it transforms the text story into prompt content suitable for generation models. Based on multi-round collaboration between a "reviser-reviewer," it optimizes the quality of prompts.
- Multimodal generation technology：
- Text generation: It generates story text based on large language models (LLMs).
- Image generation: It uses diffusion models (such as StoryDiffusion) to generate images consistent with the story content, and ensures character consistency across multiple frames through character extraction.
- Audio generation: It generates narration based on text-to-speech (TTS) models, and uses models such as AudioLDM2 or MusicGen to generate sound effects and background music.
- Video synthesis: It synthesizes the generated images, audio, and text content to form a complete narrated story video.
- Modality alignment optimization: It uses contrastive learning models (such as CLIP and CLAP) to evaluate the degree of alignment between generated content and text, and optimizes the generation results based on prompt revision mechanisms.
- Modular design: The framework supports flexible replacement of different generation models and APIs, and developers can choose more advanced models according to their needs to improve generation quality.
Project address of MM-StoryAgent
- GitHub repository：https://github.com/X-PLUG/MM_StoryAgent
- arXiv technical paper：https://arxiv.org/pdf/2503.05242
- Online demo：https://huggingface.co/spaces/wsntxxn/MM-StoryAgent
Application scenarios of MM-StoryAgent
- Children's Education and Entertainment: Generate interesting and educational audio story videos to accompany children's learning and growth.
- Digital Content Creation: Quickly generate multimodal story content for creators, reducing creation costs and improving efficiency.
- Online Education: Assist teaching by explaining knowledge in story form, enhancing learning fun.
- Multimedia Publishing: Automatically generate audio picture books, helping publishers quickly produce multimedia content.
- Smart Device Integration: Applied to smart speakers, tablets, and other devices to provide personalized story generation services.