What is FLOAT
FLOAT is an audio-driven talking head avatar generation model launched by DeepBrain AI and the Korea Advanced Institute of Science and Technology. Based on a flow matching generative model, it learns a motion latent space to achieve efficient temporally consistent motion design. The model is built on a Transformer-based vector field predictor, achieves inter-frame temporal consistency, and supports speech-driven emotion enhancement, making the generated talking motions more natural and expressive. FLOAT surpasses existing diffusion-based and non-diffusion-based methods in visual quality, motion fidelity, and generation efficiency, reaching industry-leading levels.
Main Features of FLOAT
- Audio-driven talking portrait generation: Generates talking portrait videos based on a single source image and driving audio, achieving audio-synchronized head motions, including verbal and non-verbal motions.
- Temporally consistent video generation: By modeling within the motion latent space, videos generated by FLOAT are highly temporally consistent, solving the temporal coherence issues in traditional diffusion model-based video generation.
- Emotion enhancement: Uses speech-driven emotion labels to enhance emotional expression in videos, making the generated talking motions more natural and expressive.
- Efficient sampling: Based on flow matching technology, it improves the sampling speed and efficiency of video generation.
Technical Principles of FLOAT
- Motion latent space: Shifts generative modeling from the pixel latent space to a learned motion latent space, more effectively capturing and generating temporally coherent motion.
- Flow matching: Based on flow matching, it efficiently samples in the motion latent space to generate temporally consistent motion sequences.
- Transformer-based vector field predictor: Based on a Transformer architecture, it predicts the vector field of the generated flow, and the predictor can handle frame conditions and generate temporally consistent motion.
- Frame conditioning mechanism: Based on a simple frame conditioning mechanism, it integrates driving audio and other conditions (such as emotion labels) into the generation process, achieving effective control over the motion latent space.
- Emotion control: Uses a pre-trained speech emotion predictor to generate emotion labels, feeds the labels as conditions into the vector field predictor, and introduces emotion control during the generation process.
- Fast sampling and efficient generation: Based on flow matching technology, it reduces the number of iterations in the generation process, achieving fast sampling while maintaining the high quality of generated videos.
FLOAT Project Address
- Project official website：deepbrainai-research.github.io/float
- arXiv technical paper：https://arxiv.org/pdf/2412.01064
Application Scenarios of FLOAT
- Virtual anchors and virtual assistants: In fields such as news broadcasting, weather forecasting, and online education, it generates realistic virtual anchors and provides 24-hour uninterrupted program production.
- Video conferencing and remote communication: In video conferences, it creates users' virtual avatars, enabling video communication even without a camera.
- Social media and entertainment: On social media platforms, users generate their own virtual avatars for use in live streaming, interactive entertainment, or virtual socializing.
- Games and virtual reality: In games and virtual reality applications, it is used to create or customize the facial expressions and movements of game characters, enhancing immersion.
- Film and animation production: In film post-production, generate or enhance a character's facial expressions and lip movements, reducing the need for traditional motion capture.