What is LiveCC
LiveCC is a real-time video commentary model launched by the Show Lab team at the National University of Singapore in collaboration with ByteDance, trained at scale on Automatic Speech Recognition (ASR) subtitles. LiveCC analyzes video content as quickly as a professional commentator, generating natural and fluent speech or text commentary in sync. LiveCC introduces the Live-CC-5M dataset for pre-training and the Live-WhisperX-526K dataset for high-quality supervised fine-tuning. LiveCC designed the LiveSports-3K benchmark for evaluating the model's real-time video commentary capabilities. Experiments show that LiveCC performs excellently on real-time video commentary and video question-answering tasks, demonstrating low-latency and high-quality generation capabilities.
Main Features of LiveCC
- Real-Time Video Commentary: Generates continuous, human-like real-time commentary based on video content, suitable for sports events, news broadcasting, educational videos, and many other scenarios.
- Video Question Answering: Answers questions related to video content, helping users better understand events and details in the video.
- Low-Latency Processing: Processes video streams with extremely low latency (less than 0.5 seconds per frame), supporting real-time applications.
- Multi-Scenario Adaptability: Suitable for a variety of video types, including sports, news, education, entertainment, and more.
Technical Principles of LiveCC
- Streaming Training Method: Densely interleaves Automatic Speech Recognition (ASR) words with video frames according to timestamps, enabling the model to learn time-aligned vision-language relationships. This simulates the real-time perception process of humans watching videos, allowing the model to generate commentary closely related to video content.
- Large-Scale Datasets: Builds two datasets from ASR subtitles extracted from YouTube videos: Live-CC-5M (for pre-training) and Live-WhisperX-526K (for high-quality supervised fine-tuning). The datasets provide rich training material for the model.
- Model Architecture: Based onQwen2-VLmodel architecture, combining a visual encoder and a language model to process video frames and text information. The model predicts text tokens in an autoregressive manner, taking video tokens as non-predicted input.
- Real-Time Inference: During the inference stage, the LiveCC model processes input video frame by frame, generating real-time commentary. To improve efficiency, the model caches previous prompts, visual frames, and generated text to accelerate language decoding.
- Evaluation Method: Evaluates the model's real-time commentary capability based on the LiveSports-3K benchmark, using an LLM-as-a-judge framework to compare the quality of commentary generated by different models.
LiveCC Project Address
- Project Website：https://showlab.github.io/livecc/
- GitHub Repository：https://github.com/showlab/livecc
- HuggingFace Model Library：https://huggingface.co/collections/chenjoya/livecc
- arXiv Technical Paper：https://arxiv.org/pdf/2504.16030
- Online Experience Demo：https://huggingface.co/spaces/chenjoya/LiveCC
Application Scenarios of LiveCC
- Sports Events: Provides real-time commentary and event analysis, enhancing the audience experience.
- News Reporting: Assists real-time news interpretation, improving the depth and professionalism of reporting.
- Education: Generates explanations for instructional videos, assisting skills training.
- Entertainment Media: Provides real-time plot interpretation for film and television content, increasing interactivity.
- Intelligent Assistant: Provides real-time information combined with video content, enhancing the interactive experience.