What Is StoryMem
StoryMem is an open-source video generation framework jointly released by ByteDance and Nanyang Technological University, addressing AI video generation's long-standing cross-shot consistency problem. Through an explicit visual memory mechanism, it transforms a single-shot video diffusion model into a multi-shot narrative tool. At its core is the Memory-to-Video (M2V) module, which maintains a dynamically updated keyframe memory bank and injects memory into the single-shot model to ensure cross-shot consistency.
StoryMem's Main Features
-
Multi-shot long video generation: It can generate coherent multi-shot long video stories, support narratives spanning multiple scenes, and maintain consistency of characters and scene elements.
-
Dynamic memory mechanism: By maintaining a keyframe memory bank, it injects memory into the single-shot video diffusion model, ensuring that the video remains logically and visually coherent over long time spans.
-
Cinematic image quality and high aesthetic quality: It inherits the high image quality and aesthetic characteristics of the single-shot video generation model, maintaining precise understanding of and adherence to user prompts.
-
Flexible shot control and transitions: It supports shot-level control, enabling smooth shot transitions and customized narrative applications.
-
Cross-shot consistency optimization: In multi-shot video generation, it significantly improves cross-shot consistency, with clear advantages over other methods.
-
Customized story generation: It allows users to use reference images as initial memory to generate customized stories that meet specific needs.
StoryMem's Technical Principles
-
Memory bank maintenance: After generating the first shot, the system extracts keyframe information and stores it in a dynamic memory bank for reference when generating subsequent shots.
-
Memory-to-Video (M2V) module: It injects visual features from the memory bank into the single-shot video diffusion model, fusing memory with generated content through latent space concatenation and negative RoPE offset.
-
Semantic keyframe selection strategy: After generating each shot, it uses a semantic keyframe selection strategy and aesthetic preference filtering to obtain information-rich and reliable memory frames, achieving long-term cross-shot consistency.
-
Iterative shot synthesis: By iteratively generating shots and dynamically updating the memory bank, it gradually builds coherent long video stories.
-
Lightweight LoRA fine-tuning: With only LoRA (Low-Rank Adaptation) fine-tuning, it can transform a pretrained single-shot video diffusion model into a multi-shot narrative tool.
StoryMem's Project Address
- Project official website：https://kevin-thu.github.io/StoryMem/
- Github repository：https://github.com/Kevin-thu/StoryMem
- Huggingface model library：https://huggingface.co/Kevin-thu/StoryMem
- arXiv technical paper：https://arxiv.org/pdf/2512.19539
StoryMem's Application Scenarios
-
Advertising and marketing: It quickly generates coherent advertising videos, reduces the cost of storyboard visualization, and enhances the creativity and appeal of advertising content.
-
Film and television production: It provides independent creators and small film and television teams with technical support for efficiently generating coherent narrative short films, accelerating the creative process.
-
Content creation: It helps video creators generate high-quality narrative videos in a short time, improving the efficiency and diversity of content creation.
-
Education and Training: Generate educational video stories for teaching and training scenarios, enhancing the fun and interactivity of learning.
-
Entertainment and Gaming: Provide customized story generation tools for game developers and the entertainment industry, enriching game plots and entertainment content.
-
Social Media: Provide social media creators with tools to quickly generate engaging video stories, increasing content virality and user engagement.