What Is InstructAV2AV
InstructAV2AV is an audio-video joint editing model jointly launched by the Beijing Academy of Artificial Intelligence and Peking University. Users only need a single natural language instruction to simultaneously edit the video frames and the corresponding sound during end-to-end generation, without the need for manual masks or step-by-step processing. The model supports dialogue modification, character replacement, and object insertion and deletion, while keeping the background and ambient sound unchanged and maintaining audio-visual temporal synchronization.
InstructAV2AV's Main Features
-
Identity-Preserving Speech Modification: Replaces only a person's spoken content while preserving their appearance.
-
Audio-Video Instance Replacement: Replaces a specified person in the frame together with their voice and dialogue.
-
Instance Insertion: Adds a new object to the frame and automatically generates synchronized sound for it.
-
Instance Removal: Deletes a specified object and its corresponding sound in the audio track.
-
Attribute Composition Editing: A person's appearance, spoken content, and voice timbre can be modified independently and freely combined.
InstructAV2AV's Technical Principles
-
Source Latent Concatenation: Concatenates the latents of the source video and source audio with noise along the channel dimension, making the source content a structural condition throughout the denoising process, constraining the model to preserve the background, irrelevant objects, and ambient sound, and avoiding "changing one target while redrawing the entire world."
-
SIGA Gated Attention: Source-Instruction Gated Attention lets each token interact with both source features and instruction features simultaneously, using learnable soft gating to decide position by position how much to change and how much to keep, achieving a dynamic balance between content preservation and instruction execution.
-
Dual-Stream Interactive Architecture: Based on the Ovi symmetric dual-stream diffusion Transformer, the video and audio branches each model self-attention before exchanging features through bidirectional cross-modal attention, so that visual motion and sound events mutually constrain each other and remain synchronized.
-
Two-Stage Training: Cross-modal attention is disabled to train unimodal editing capabilities separately, then the full architecture is restored for joint fine-tuning to learn fine-grained correspondences between sound and picture.
-
Automated Data Pipeline: Builds the InsAVE-80K dataset through a three-stage process of material filtering, editing target synthesis, five-dimensional automatic evaluation plus manual review, solving the problem of scarce paired training data.
Follow WeChat and reply "open source" to join theAI Open Source Project Communication Group
How to Use InstructAV2AV
-
Clone the Code: Clone the InstructAV2AV code repository from GitHub to your local machine.
-
Configure the Environment: Configure the Python environment and install the required dependencies according to the repository instructions.
-
Download the Weights: Download the InstructAV2AV model weights from Hugging Face.
-
Prepare the Material: Prepare a video clip with sound as input.
-
Input the Instruction: Enter a natural language editing instruction describing what you want to modify.
-
Run Inference: Run the inference script and wait for the model to generate the edited video and audio.
-
Save the Result: View the output result and save the edited audio-video file.
InstructAV2AV project address
- Project website：https://hjzheng.net/projects/InstructAV2AV/
- GitHub repository：https://github.com/suimuc/InstructAV2AV
- HuggingFace model library：https://huggingface.co/suimu/InstructAV2AV
- arXiv technical paper：https://arxiv.org/pdf/2605.18467
Comparison of InstructAV2AV with similar competing products
| Comparison item | InstructAV2AV | Ovi |
|---|---|---|
| Product positioning | Instruction-driven joint audio-video editing model | Joint audio-video generation model |
| Core capability | Modify the visuals and sound based on the source video | Generate video and audio from scratch |
| Input method | Source video + source audio + natural language instruction | Text prompt (generate from scratch) |
| Background preservation | Accurately preserve irrelevant background and ambient sound | Does not involve preserving source content |
| Audio-visual synchronization | Bidirectional cross-modal attention ensures synchronization after editing | Synchronized during generation, does not support editing scenarios |
| Technical architecture | Built upon the Ovi dual-stream diffusion Transformer, with source latent concatenation and SIGA gating added | Symmetric dual-stream diffusion Transformer |
| Applicable scenarios | Dialogue modification, character replacement, object insertion and removal | Creative text-to-audio-video generation |
Application scenarios of InstructAV2AV
-
Film and television post-production: Film and television post-production personnel use it to replace actors' dialogue and synchronize lip movements, reducing reshoot costs.
-
Short video production: Short video creators use it to modify the characters and sounds in footage with a single sentence, improving creative efficiency.
-
Virtual humans: Virtual human operations teams use it to adjust the appearance, voice, and spoken content of digital humans.
-
Advertising creativity: Advertising professionals use it to batch-generate promotional videos with different character and dialogue versions.
-
Interactive content production: Game and education professionals use it to dynamically generate multimedia materials with consistent audio and visuals.