What is StoryTeller
StoryTeller is a system jointly launched by ByteDance, Shanghai Jiao Tong University, and Peking University. It can improve the quality and consistency of long video descriptions based on audio-visual character recognition technology. The system combines low-level visual concepts and high-level plot information to generate detailed and coherent video descriptions. StoryTeller consists of three main modules: video segmentation, audio-visual character recognition, and description generation. It can effectively process videos several minutes long and demonstrates higher accuracy than existing models on the MovieQA task, with 9.5% higher accuracy than the strongest baseline, Gemini-1.5-pro.
Main features of StoryTeller
- Video segmentation: Splits long videos into multiple short segments, maintaining the independence and completeness of each segment.
- Audio-visual character recognition: Combines audio and visual information to identify the characters corresponding to dialogue in the video.
- Description generation: Generates detailed descriptions for each video segment and integrates them into a coherent narrative for the entire long video.
- Dataset construction: Creates and uses the MovieStory101 dataset, providing training and testing data for long video description.
- Automatic evaluation: Based on MovieQA, uses GPT-4 to automatically evaluate the accuracy and quality of video descriptions.
- Model training and fine-tuning: Trains multimodal large language models to improve the accuracy of character recognition and video description.
- Global decoding: Ensures that the same character maintains consistent recognition results across different video segments.
Technical principles of StoryTeller
- Multimodal fusion: Integrates visual (video frames), audio (dialogue), and textual (subtitles and descriptions) information to comprehensively understand video content.
- Audio separation and character ID assignment: Uses an audio embedding model to embed each dialogue, assigns global IDs based on a clustering algorithm, and assigns the same ID to similar audio embeddings to indicate the same character.
- Audio-visual character recognition model: Uses a large language model (such as Tarsier-7B) combined with OpenAI's Whisper-large-v2 audio encoder to map each audio ID to a specific character.
- Global decoding algorithm: During inference, ensures that the global IDs of the same character in different segments are mapped to consistent character names, improving the accuracy of character recognition.
- Video description generation: Uses recognition results as input, generates detailed descriptions for each segment based on a large language model, and integrates them into a complete video description.
StoryTeller project address
- GitHub repository：https://github.com/hyc2026/StoryTeller
- arXiv technical paper：https://arxiv.org/pdf/2411.07076
Application scenarios of StoryTeller
- Film and video content production: Automatically generates descriptions of movie trailers or movie clips, helping directors and screenwriters quickly understand video content. Assists video editing work by quickly locating key segments in videos based on video descriptions.
- Video content analysis: In the field of video analysis, extracts key information from video content, such as characters, plot, and actions, for in-depth content analysis.
- Assisting visually impaired individuals: Provides audio descriptions of video content for visually impaired individuals, helping them better understand video content and storylines.
- Education and training: In the field of education, provide detailed descriptions of video teaching materials for students to enhance the learning experience. In vocational training, generate detailed step descriptions for video tutorials to improve training efficiency.
- Video search and indexing: Improve the accuracy of video search, and quickly retrieve relevant segments in videos based on video descriptions.