Vidi2 - A multimodal video understanding and generation model launched by ByteDance

What is Vidi2

Vidi2 is a multimodal large language model launched by ByteDance that focuses on video understanding and creation. It has reached industry-leading levels in multimodal temporal retrieval (TR) and has made significant progress in spatiotemporal grounding (STG) and video question answering (Video QA). Based on text queries, Vidi2 can not only identify the corresponding timestamps in a video, but also mark the bounding boxes of target objects. Vidi2 introduces new benchmarks VUE-STG and VUE-TR-V2 to better evaluate STG capabilities. In practical applications, Vidi2 supports functions such as intelligent editing, intelligent storyboarding, and intelligent subtitles, helping creators carry out video creation more efficiently.

Main features of Vidi2

• Video understanding and retrieval: It can process hours-long video footage and quickly locate relevant time segments in the video based on text queries, supporting long-video understanding in complex scenarios.
• Spatiotemporal grounding (STG): It can not only identify timestamps in a video, but also mark the bounding boxes of target objects within these time segments, achieving precise spatiotemporal grounding.
• Video question answering (Video QA): It supports video question answering tasks and can answer users' questions based on video content, providing more comprehensive multimodal reasoning capabilities.
• Intelligent creation tools: It supports multiple functions such as automatic editing, intelligent storyboarding, and intelligent subtitles, helping creators quickly transform long video footage into short videos suitable for platforms such as TikTok.
• Multi-view switching and intelligent reframing: It supports automatic multi-view switching and intelligent, composition-aware reframing and cropping, improving the visual effect of videos.

Technical principles of Vidi2

• Multimodal fusion: Vidi2 combines a visual encoder (used to process video frames and clips) with a large language model, forming integrated long-video understanding and expression capabilities through instruction fine-tuning and alignment.
• Spatiotemporal grounding (STG): Through spatiotemporal grounding technology, Vidi2 can precisely locate relevant timestamps and the bounding boxes of target objects in a video based on text queries. Based on multimodal feature extraction and similarity calculation, it processes the spatiotemporal information of videos through deep learning models (such as Transformer).
• Video understanding and retrieval: Vidi2 uses multimodal text-video retrieval technology to convert text and video data into vector representations, and then implements text-to-video retrieval through similarity calculation (such as cosine similarity). Through feature extraction and deep learning technology, it performs structured analysis of video content to achieve efficient video retrieval.
• Multi-granularity temporal modeling: Vidi2 adopts multi-granularity temporal modeling technology, which can process information at different time scales from a single frame to long video clips, and combines memory and retrieval mechanisms (such as sliding window + memory token/retrieval augmentation) to achieve efficient understanding and retrieval of long videos.
• Cross-modal alignment: Through cross-modal alignment technology, Vidi2 effectively aligns the visual information, audio information, and text information in videos, thereby achieving more comprehensive video content understanding.

Project address of Vidi2

• Project official website：https://bytedance.github.io/vidi-website/
• Github repository：https://github.com/bytedance/vidi
• arXiv technical paper：https://arxiv.org/pdf/2511.19529

Application scenarios of Vidi2

• Intelligent video editing: Automatically extract highlights from long videos and generate content suitable for short video platforms, improving creation efficiency.
• Video Question Answering: Answer user questions based on video content, enhancing the interactive experience between users and video content.
• Spatiotemporal Localization-Assisted Editing: By precisely locating the time and objects in a video, assist creators with complex editing tasks such as multi-view switching and intelligent recomposition.
• Automatic Subtitle Generation: Automatically generate subtitles for videos, improving video accessibility and user experience.
• Content Creation Assistance: Help creators turn simple prompts or topics into complete video scripts, including titles, hooks, and storyboards.