SEED-Story - A multimodal story generation model launched by Tencent in collaboration with HKUST and CUHK

What is SEED-Story

SEED-Story is a multimodal story generation model launched by Tencent in collaboration with the Hong Kong University of Science and Technology and the Chinese University of Hong Kong. Based on a multimodal large language model (MLLM), it can predict text and visual tokens and generate stylistically consistent images through a visual de-tokenizer. SEED-Story introduces a multimodal attention mechanism and supports the generation of coherent stories up to 25 sequences long. SEED-Story also provides StoryStream, a large-scale high-resolution dataset for model training and evaluation.

Main features of SEED-Story

• Multimodal story generation: SEED-Story can generate long stories containing coherent narrative text and stylistically consistent images based on a given starting image and text.
• Multimodal attention sink mechanism: SEED-Story proposes a multimodal attention sink mechanism to generate stories in an efficient autoregressive manner, and can generate stories up to 25 sequences long.
• Large-scale dataset : SEED-Story released a large-scale, high-resolution dataset named StoryStream, used to train models and quantitatively evaluate multimodal story generation tasks from different aspects.
• Story instruction tuning: During story generation, SEED-Story fine-tunes the model through an instruction tuning process, enabling it to predict the next image and next sentence of the story.
• Visualization and interaction: SEED-Story also provides visual comparisons, showing that the generated images have higher quality and better consistency compared with other baseline models.

Technical principles of SEED-Story

• Multimodal large language model (MLLM)：Uses large language models to understand, generate, and predict text and visual tokens.This type of model can process and generate text data while also understanding and generating visual content.
• Text and visual token prediction：The SEED-Story model can predict text tokens (elements or words in text) and visual tokens (elements in images).
• Visual de-tokenizer：Converts text and visual tokens generated by the language model into images, generating consistent and stylized images through the visual de-tokenizer.
• Multimodal attention mechanism：Introduces a multimodal attention mechanism, enabling the model to focus on the interrelationships between text and images when generating stories.This mechanism allows the model to handle long-sequence generation tasks more effectively.
• Autoregressive generation：The model generates stories in an autoregressive manner, meaning each newly generated token depends on all previously generated tokens, in order to maintain story coherence.
• Long-sequence generation capability：Through the multimodal attention mechanism, SEED-Story can generate story sequences longer than the training sequences.
• StoryStream dataset：Provides a large-scale high-resolution dataset for training models and evaluating multimodal story generation tasks.The dataset contains visually engaging high-resolution images and detailed narrative text.
• Training process：Includes pre-training the de-tokenizer to reconstruct images, sampling interleaved image-text sequences for training, and using the MLLM's regression image features to adjust the generated images.

SEED-Story project address

Application Scenarios of SEED-Story

• Creative Writing Assistance: Helps writers and content creators generate story outlines, character descriptions, and plot development, inspiring creative inspiration.
• Education and Learning: As a teaching tool, it helps students understand narrative structures and improve their writing and creative thinking abilities.
• Entertainment and Game Development: Generates dynamic storylines and character backgrounds in video games, providing an immersive experience.
• Advertising and Marketing: Quickly generates engaging advertising stories or marketing materials, improving the appeal and personalization of advertising content.
• Film and Animation Production: Assists scriptwriting, generates character dialogue and scene descriptions, and accelerates the production process.