What is Spirit LM
Spirit LM is a multimodal language model launched by the Meta AI team that can seamlessly mix text and speech data. Spirit LM is based on a pretrained text language model and is extended to the speech modality through continued training on text and speech units. The model has two versions: the base version (BASE) and the expressive version (EXPRESSIVE). The base version uses speech semantic units, while the expressive version, in addition to semantic units, also uses pitch and style units to model the expressiveness of speech. During training, Spirit LM concatenates speech and text sequences into a single token set, based on a word-level interleaving method. It can generate text with the semantic capabilities of a text model and generate speech with the expressive capabilities of a speech model. Spirit LM can learn new tasks across modalities with few samples, such as automatic speech recognition (ASR), text-to-speech (TTS), and speech classification.
Main features of Spirit LM
- Cross-modal language generation: Spirit LM can generate text and speech, enabling seamless switching.
- Semantics and expressiveness: Combines the semantic capabilities of a text model with the expressive capabilities of a speech model.
- Few-shot learning: Quickly learns new tasks from a small number of samples, such as ASR, TTS, and speech classification.
- Emotion preservation: The expressive version (EXPRESSIVE) can understand and generate speech and text with specific emotions.
- Multimodal understanding: Understands and generates cross-modal content, for example converting text to speech or vice versa.
Technical principles of Spirit LM
- Pretraining and extension: Based on a pretrained text language model, continued training on text and speech units is used to extend the model's speech processing capabilities.
- Interleaved training: Uses a word-level interleaving method to concatenate speech and text sequences into a single token set for training, achieving alignment between speech and text.
- Dual-modal tokens：
- Base version (BASE): Uses speech semantic units (such as HuBERT tokens).
- Expressive version (EXPRESSIVE): In addition to semantic units, combines pitch (F0) and style units to capture the expressiveness of speech.
- Encoding and decoding：
- Encoder: Converts speech into tokens, using models such as HuBERT for speech encoding.
- Decoder: Restores tokens to the original modality (text or speech).
- Datasets and training：
- Uses large-scale text datasets and speech datasets.
- Performs interleaved training based on aligned speech + text datasets.
Spirit LM project addresses
- Project official website：speechbot.github.io/spiritlm
- GitHub repository：https://github.com/facebookresearch/spiritlm
- arXiv technical paper：https://arxiv.org/pdf/2402.05755
Application scenarios of Spirit LM
- Voice assistants: Integrated into smart devices to provide voice interaction functions, such as answering questions, setting reminders, and controlling smart home devices.
- Automatic speech recognition (ASR): Converts speech into text, used in speech transcription, meeting minutes, and voice command systems.
- Text-to-speech (TTS): Converts text information into speech output, used in audiobooks, news broadcasts, and navigation systems.
- Emotion analysis: Analyze sentiment tendencies in speech or text, used in customer service, market research, and social media monitoring.
- Speech Translation: Enable real-time translation of speech input, helping cross-language communication, suitable for international conferences and travel scenarios.