musiclm-pytorch is an implementation of the Google MusicLM model in PyTorch. MusicLM is a SOTA model proposed by Google for music generation using attention networks. According to the repository description, MusicLM basically uses a text-conditioned AudioLM, but unexpectedly uses embeddings generated by a text-audio contrastive learning model called MuLan. MuLan will be built in this repository, while AudioLM is modified from another repository to support the music generation needs here.
Main capabilities and implementation approach:
- Build the MuLan text-audio contrastive learning model for generating conditional embeddings.
- Modify AudioLM to accept conditional embeddings, and optionally handle different dimensions through a separate projection.
- Combine AudioLM and MuLan into MusicLM for generation, and use MuLan for filtering.
- Provide dynamic positional bias for self-attention in AST.
- Implement MusicLM to generate multiple samples and use MuLan to select the best match.
- Support variable-length audio in the audio transformer through masking.
- Add a MuLan version for open clip.
- Set all appropriate spectrogram hyperparameters.
Usage:
- MuLaN first needs to be trained. To obtain the conditional embeddings for the three transformers in AudioLM, MuLaNEmbedQuantizer must be used.
- To train (or fine-tune) the three transformers in AudioLM, follow the training instructions in audiolm-pytorch, but pass a MulanEmbedQuantizer instance to the training class through the keyword audio_conditioner, for example SemanticTransformerTrainer.
- After sufficiently training all three transformers (semantic, coarse, fine), pass the fine-tuned or from-scratch trained AudioLM and MuLaN (wrapped in MuLaNEmbedQuantizer) to MusicLM.
Applicable scenarios: This repository is aimed at researchers and developers interested in music generation, text-conditioned audio generation, and reproduction of related deep learning models. The repository mentions that if you are interested in helping with LAION community for reproduction, you can join.
Limitations and notes: The repository does not provide specific installation commands, dependency versions, pretrained model download links, generation effect examples, fee or price information, platform support, or detailed usage steps. The above content has not yet been provided.
