MuseV is an open-source virtual human video generation framework based on diffusion models by Tencent Music Entertainment's Lyra Lab. Its core feature is the use of a novel "Visual Conditioned Parallel Denoising" approach to achieve infinite-length, high-fidelity virtual human video generation.
Purpose MuseV is used to generate virtual human videos, covering three types of tasks: Image2Video, Text2Image2Video, and Video2Video. It can be combined with MuseTalk (a real-time high-quality lip-sync model) to form a complete virtual human generation solution; the official release also includes MusePose (a pose-controlled image-to-video generation framework), with the hope that the community will jointly advance the vision of end-to-end virtual human generation with full-body motion and interaction capabilities.
Main capabilities
- Supports infinite-length generation: uses the Visual Conditioned Parallel Denoising approach.
- Provides virtual human video generation checkpoints trained on human datasets.
- Task types: Image2Video, Text2Image2Video, Video2Video.
- Compatible with the Stable Diffusion ecosystem: base_model, lora, controlnet, etc.
- Supports multiple reference image techniques: IPAdapter, ReferenceOnly, ReferenceNet, IPAdapterFaceID.
- Officials state that the training code will be released soon.
- Provides a Gradio script to generate a GUI locally for video generation.
Models and versions The official release includes the MuseV project and the trained models musev and muse_referencenet (March 27, 2024), followed by the addition of a huggingface space gradio to generate videos in the GUI (March 30, 2024). Important bug fix notice: the model_name of unet and the ip_adapter of Command in musev_referencenet_pose are incorrect; musev_referencenet_pose should be used instead of musev_referencenet.
Usage Environment preparation: Officials recommend prioritizing docker to prepare the Python environment, the default conda environment name is musev; a conda environment can also be created from environment.yaml, and if Docker is not used, the mmlab packages need to be installed additionally. Additional dependencies such as diffusers, controlnet_aux, and mmcm need to be installed. The project also mentions a Windows all-in-one package (third-party integration, not officially verified, maintained, or updated).
Custom packages: the project includes modified diffusers and controlnet_aux, as well as the self-developed MMCM (multimedia, cross-modal processing package).
Model download: motion is a text2video model trained on the tiny ucf101 and tiny webvid datasets (about 60,000 video-text pairs); muse v/unet trains only the unet motion module, with about 8G of VRAM; musev_referencenet trains unet, referencenet, and IPAdapter, with about 12G of VRAM; musev_referencenet_pose fixes referencenet and controlnet_pose on the basis of musev_referencenet, and trains unet motion and IPAdapter, with about 12G of VRAM; t2i/sd1.5 is a text2image model, and different t2i base_model choices have a significant impact, with examples including majicmixRealv6Fp16 and fantasticmix_v10; IP-Adapter/models includes image_encoder, ip-adapter_sd15.bin, ip-adapter-faceid_sd15.bin, etc.
Configuration and long video generation: the model path and abbreviation need to be set in the configuration, involving T2I_all_model.py, motion_model.py, and example.yaml. Commonly used parameters include time_size, n_batch, context_frames, etc. There are two ways to generate long videos: parallel denoising with visual conditions (set n_batch=1, and set time_size to the desired total number of frames); traditional end-to-end (time_size=context_frames=the number of frames in one shot, 12, context_overlap=0). Model parameters support referencenet, IPAdapter, IPAdapterFaceID, and Facein, and also include parameters such as video_guidance_scale, use_condition_image, redraw_condition_image, and video_negative_prompt that affect motion magnitude and generation results.
Special Video2Video parameters include setting video_path as the reference video (supporting rgb video and controlnet_middle_video), as well as which2video, controlnet_name, video_is_middle, video_has_condition, etc. musev_referencenet_pose is only used for pose2video, is trained based on musev_referencenet, has only the motion module and no referencenet, and requires less VRAM.
Applicable scenarios Suitable for research and creative scenarios such as virtual human video generation, image-driven character animation, text-to-virtual-human video, video style or motion transfer, and pose-driven video generation, and can also be combined with MuseTalk and MusePose for exploring end-to-end digital human solutions.
Limitations The official description clearly lists multiple limitations: insufficient generalization ability, with some visual condition images performing well and others poorly, and some t2i pretrained models performing well and others poorly; limited video generation types and motion range, partly due to training data limitations; the released MuseV was trained on about 60,000 human text-video pairs at 512*320 resolution, with a larger motion range at lower resolution but lower image quality, and a smaller motion range at higher image quality; watermarks may appear due to webvid; long video generation types are limited, and although parallel denoising with visual conditions can address cumulative errors, the current method is only suitable for relatively fixed shot scenarios; referencenet and IP-Adapter are insufficiently trained; the code structure is not regular enough, and the supported features are rich and dynamic but complex and not refactored, requiring time to become familiar with.
License The code is open-sourced under the MIT license, with no restrictions for academic or commercial use; the trained models may only be used for non-commercial research purposes; other open-source models used must follow their respective licenses, such as insightface, IP-Adapter, ft-mse-vae, etc.; the test data comes from the internet,and may only be used for non-commercial research purposes. Regarding AIGC, the project encourages a positive impact on the field of AI video generation, and users are free to use the tool to create videos, but must comply with local laws and use it responsibly; the developers are not responsible for potential misuse.
