What is Deep Video Discovery
Deep Video Discovery (DVD) is a deep video discovery agent launched by Microsoft, designed specifically for understanding and analyzing long videos. Deep Video Discovery divides long videos into multiple shorter clips and, based on the advanced reasoning capabilities of large language models (LLMs), autonomously plans and selects appropriate tools and parameters to collect information. Deep Video Discovery is equipped with a set of search-centric tools, including global browsing, clip search, and frame inspection, and can collect information at different levels, gradually building an understanding of video content based on iterative reasoning. In multiple long video understanding benchmarks, Deep Video Discovery has achieved state-of-the-art performance, significantly improving the accuracy and efficiency of long video understanding.
Main Features of Deep Video Discovery
- Multi-granularity video understanding: Understands and analyzes video content from three levels: global, clip, and frame, providing comprehensive video understanding capabilities.
- Autonomous search and reasoning: Autonomously plans and executes search strategies, dynamically selects appropriate tools and parameters according to user queries, gradually collects information, and builds an understanding of video content.
- Efficient information retrieval: Based on tools such as global browsing, clip search, and frame inspection, quickly locates and extracts video clips and details relevant to user queries.
- Long video understanding: Excels at processing information-intensive videos that are several hours long, and can effectively address the temporal and spatial complexity issues in long video understanding.
- Flexible tool use: Flexibly combines and uses multiple tools according to different task requirements, achieving efficient video content analysis and question answering.
Technical Principles of Deep Video Discovery
- Multi-granularity video database construction: Evenly divides long videos into multiple shorter clips, each about 5 seconds. Extracts video information from three levels: global, clip, and frame. The global layer provides the subject and event summaries of the video, the clip layer provides text descriptions (caption), and the frame layer retains original pixel information. Constructs a structured database containing decoded frames, text descriptions, and corresponding embedding vectors, supporting fast retrieval and detailed analysis.
- Autonomous search and answer generation：
- Global Browse: Provides a global summary of the video, helping the agent quickly understand the video's subject and main events.
- Clip Search: Based on text embedding matching, quickly retrieves video clips relevant to user queries.
- Frame Inspect: Performs fine-grained visual question answering (VQA) within a specific time range, extracting frame-level detailed information.
- Autonomous agent design: The Agent is based on an iterative observation-reasoning-action loop, uses the reasoning capabilities of LLMs, dynamically selects and uses tools, gradually collects information, and builds an understanding of video content.
- Iterative reasoning: The Agent selects appropriate tools and parameters according to the current observation state and reasoning results, gradually refines the query, and ultimately generates accurate answers.
- LLM-driven reasoning: LLM serves as the core, responsible for reasoning and planning. Based on the current conversation history and observations, the LLM selects appropriate tools and parameters and dynamically adjusts its reasoning strategy. According to task requirements, it flexibly combines and uses different tools to build multi-step tool-use chains and solve complex query tasks.
Deep Video Discovery project address
- arXiv technical paper：https://arxiv.org/pdf/2505.18079
Application scenarios of Deep Video Discovery
- Education: Online education platforms analyze long video courses, allowing students to quickly locate specific knowledge points or chapters in the course.
- Sports event analysis: Sports event analysis of match videos quickly extracts key events.
- Video surveillance: In security surveillance systems, real-time analysis of surveillance video quickly identifies abnormal behavior or events.
- Film and television production: Film and television post-production teams analyze footage and quickly find the required shots.
- Enterprise meeting records: Enterprises analyze meeting videos and quickly extract meeting highlights and key decisions.