infinitetalk

InfiniteTalk - Audio-Driven Video Generation

Task Objective

• This Skill is used to: convert audio (speech) into synchronized speaker video, supporting the generation of audio-driven talking videos from a single image or an existing video
• Capabilities include:
  • Image-to-Video: Generate an audio-driven talking video from a single image
  • Video-to-Video: Perform audio-driven redubbing on an existing video
  • Multi-dimensional synchronization: Precise alignment of lip shape, head movement, body posture, and facial expressions with audio
  • Unlimited duration: Supports video generation of unrestricted duration
  • Low VRAM adaptation: Supports VRAM optimization solutions such as quantization and model offloading
• Trigger conditions: Used when there is a need to generate audio-driven digital human videos, video voiceovers, or virtual anchor content

Prerequisites

• Model download: Before using this Skill, the required model weight files must first be downloaded. For specific steps, see references/model_download.md
• Hardware requirements:
  • GPU: A GPU with 16GB+ VRAM is recommended (quantization solutions can be used to adapt to low-VRAM devices)
  • Memory: 32GB+ system memory is recommended
  • Disk space: At least 50GB of available space (model weights are about 30GB)
• Environment configuration: For detailed dependency installation, see references/environment_setup.md

Operation Steps

Mode 1: Image-to-Video (Generate Video from Image)

1. Prepare Input

   • Ensure there is a clear face image as input
   • Prepare an audio file (supports formats such as mp3 and wav)
   • Optional: Use the TTS function to generate audio from text

2. Execute Generation

   • Call scripts/infer_infinitetalk.py for inference
   • Parameter description:
     • input_path: Input image path
     • audio_path: Driving audio path (or provide text to use TTS)
     • output_path: Output video path
     • mode: clip(single segment) or streaming(long video)
     • size: infinitetalk-480(480P) or infinitetalk-720（720P）
     • sample_steps: Number of sampling steps (default 40)
     • sample_audio_guide_scale: Audio guidance strength (default 4.0)

3. Verify Output

   • Check whether the generated video is well synchronized
   • Confirm that lip shape and head movements match the audio
   • If there are anomalies, adjust the sample_audio_guide_scale parameter

Mode 2: Video-to-Video (Video Redubbing)

1. Prepare Input

   • Prepare a reference video file
   • Prepare the target audio file

2. Execute Generation

   • Use the same script, but input_path points to the video file
   • The script will automatically extract the first frame of the video as a reference

3. Process Long Videos

   • Use streaming mode to generate videos of unlimited duration
   • Control the driving frame length through the motion_frame parameter (default 9)

Mode 3: Generate Audio Using TTS

1. Text-to-Speech

   • Provide the text content to be synthesized
   • Specify the voice model (Kokoro-82M)
   • The script will automatically generate the audio file

2. Generate Video

   • Use the generated audio to drive video generation
   • Supports two-person dialogue mode (use markers (s1) and (s2) to distinguish speakers)

Resource Index

• Core script: see scripts/infer_infinitetalk.py(audio-driven video generation inference)
• Environment configuration: see references/environment_setup.md(dependency installation and system configuration)
• Model download: see references/model_download.md(model weight download guide)
• Usage examples: see references/usage_examples.md(typical scenarios and command examples)

Notes

• The model weights are large (about 30GB), so they need to be downloaded in advance for first-time use
• It is recommended to use a GPU with high VRAM (16GB+); low-VRAM devices can use quantization solutions
• The input audio is recommended to have a sampling rate of 16000Hz; the script will automatically perform loudness normalization
• The input image/video should contain a clear face region
• Generation speed depends on GPU performance; at 480P resolution, generating 1 second of video takes about 5-10 seconds
• If you encounter an out-of-memory error, you can try:
  • Lower the resolution (use size=infinitetalk-480）
  • Enable quantization (add --quant int8 parameter)
  • Enable model offloading (add --offload_model true）

Usage Examples

• Basic image-to-video generation:

  python scripts/infer_infinitetalk.py \
    --input_path ./input.jpg \
    --audio_path ./audio.wav \
    --output_path ./output.mp4 \
    --size infinitetalk-480 \
    --mode clip
  

• Long video generation:

  python scripts/infer_infinitetalk.py \
    --input_path ./input.jpg \
    --audio_path ./long_audio.wav \
    --output_path ./long_output.mp4 \
    --size infinitetalk-480 \
    --mode streaming
  

• Generate using TTS:

  python scripts/infer_infinitetalk.py \
    --input_path ./input.jpg \
    --text "你好，今天天气真不错" \
    --output_path ./tts_output.mp4 \
    --size infinitetalk-480