What is INFP
INFP is an audio-driven head generation framework designed specifically for two-person conversational interactions. It can automatically switch roles under the guidance of conversational audio, without the need for manual role assignment and role switching. INFP consists of two stages: a motion-based head imitation stage and an audio-guided motion generation stage. Through experiments and visualizations, the excellent performance and effectiveness of the INFP method are verified. INFP proposes a large-scale two-person conversation dataset, DyConv, to support progress in this research field.
Main Features of INFP
- Automatic Role Switching: In two-person conversations, INFP can automatically switch roles without the need for manual role assignment and role switching, enhancing the naturalness and smoothness of interaction.
- Lightweight and Efficient: While maintaining powerful functionality, INFP is also lightweight. It can achieve inference speeds of over 40 fps on an Nvidia Tesla A10, which means INFP can support real-time intelligent agent interactions, whether it is communication between agents or interaction between humans and agents.
- Interactive Head Generation: INFP includes two key stages: motion-based head imitation and audio-guided motion generation. The first stage encodes facial communication behaviors from real conversation videos into a low-dimensional motion latent space, while the second stage maps the input audio to these motion latent codes, achieving audio-driven head generation.
- Large-Scale Two-Person Conversation Dataset DyConv: To support progress in this research field, INFP proposes a large-scale two-person conversation dataset, DyConv, consisting of rich dyadic conversations collected from the internet.
Technical Principles of INFP
- Motion-Based Head Imitation Stage: In this stage, the framework learns to project facial communication behaviors from real-life conversation videos into a low-dimensional motion latent space. This process involves extracting facial communication behaviors from a large number of real conversation videos and encoding them into motion latent codes that can drive the animation of static images.
- Audio-Guided Motion Generation Stage: In the second stage, the framework learns the mapping from input two-channel audio to motion latent codes. This stage is implemented through a denoising process, thereby achieving audio-driven head generation in interactive scenarios.
- Real-Time Interaction and Style Control: INFP supports real-time interaction, allowing users to interrupt or respond to the virtual avatar at any time during the conversation. By extracting style vectors from arbitrary portrait videos, INFP can also globally control the emotion or attitude in the generated results.
INFP Project Address
- Project Official Website：https://grisoon.github.io/INFP/
- arXiv Technical Paper：https://www.arxiv.org/pdf/2412.04037
Application Scenarios of INFP
- Video Conferencing and Virtual Assistants: The INFP framework can achieve realism, interactivity, and real-time performance, making it suitable for real-time scenarios such as video conferencing and virtual assistants, providing a more natural and smooth interactive experience.
- Social Media and Interactive Entertainment: On social media platforms or in interactive entertainment applications, INFP can be used to generate interactive avatars with natural expressions and head movements, enhancing the user's interactive experience.
- Education and Training: INFP can be used to create virtual teachers or trainers, providing a more vivid and interactive teaching experience.
- Customer Service: In the field of customer service, INFP can be used to generate virtual customer service representatives, providing more humanized service.
- Advertising and Marketing: INFP can be used to generate more attractive virtual spokespersons for advertising and marketing campaigns, providing a more realistic and interactive advertising experience.
- Games and Simulation: In gaming and simulation environments, INFP can be used to create more realistic and interactive characters, enhancing the immersion and interactivity of games.