SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Llasa TTS - An Open-Source Text-to-Speech Model from The Hong Kong University of Science and Technology

Author: AI小集 · Original source: AI工具集

What is Llasa TTS

Llasa TTS is an open-source text-to-speech (TTS) model launched by The Hong Kong University of Science and Technology based on the LLaMA architecture, supporting high-quality speech synthesis and cloning. Llasa TTS is based on a single-layer vector quantization (VQ) codec and a single Transformer architecture, fully aligned with the standard LLaMA model, generating natural and fluent speech, supporting emotional expression andvoice cloningand other functions. Llasa TTS performs excellently in both training and inference stages, improving the naturalness, prosody accuracy, and emotional expression ability of speech based on the computational resources of extended training time and inference time. Llasa TTS provides models with 1B, 3B, and 8B parameter scales, supporting multilingual synthesis.

Llasa TTS

Main Features of Llasa TTS

  • High-Quality Speech Synthesis: Generates natural and fluent speech, supporting both Chinese and English, suitable for various application scenarios.
  • Emotional Expression: Injects emotional information to generate speech with emotional colors such as happiness, anger, sadness, etc., enhancing the naturalness and expressiveness of the speech.
  • Voice Cloning: Requires only a small number of audio samples (e.g., 15 seconds) to clone the timbre and emotion of a specific voice, achieving personalized speech synthesis.
  • Long Text Support: Supports processing long text inputs and generating coherent speech output, suitable for scenarios such as audiobooks and voice announcements.
  • Zero-Shot Learning: Supports speech synthesis for unseen speakers or emotions without additional fine-tuning.

Technical Principles of Llasa TTS

  • Transformer-Based Architecture: Based on a single Transformer architecture, fully aligned with standard large language models. Uses a single-layer vector quantization (VQ) codec to convert speech waveforms into discrete speech tokens, modeled based on Transformer.
  • Speech Tokenizer：
    • Encoding: Decomposes speech signals into semantic features and acoustic features, extracted based on pre-trained Wav2Vec2-BERT and convolutional modules respectively.
    • Quantization: Uses improved vector quantization (VQ) technology to encode features into discrete tokens.
    • Decoding: Decodes discrete tokens back into high-quality speech waveforms, supporting the reconstruction of semantic and acoustic information.
  • Training and Inference Scaling：
    • Training Time Scaling: Based on increasing model scale (e.g., 1B, 3B, 8B parameters) or training data volume (e.g., 250k hours of speech data), improves speech naturalness and prosody accuracy.
    • Inference Time Scaling: Introduces a speech understanding model as a verifier during the inference stage, using complex search strategies (e.g., beam search, best candidate selection) to optimize generation results, enhancing emotional expression and timbre consistency.
  • Autoregressive Generation: Based on an autoregressive generation method, generates speech tokens one by one, ensuring that the generated speech is consistent with the input text in semantics and prosody.

Llasa TTS Project Address

Application Scenarios of Llasa TTS

  • Intelligent Voice Assistant: Provides natural and fluent voice interaction functions for smart devices or software, improving user experience.
  • Audiobooks and Online Education: Convert text content into vivid speech, providing users or students with an auditory learning experience.
  • Voice Announcements and Customer Service: Used for news broadcasting, traffic information prompts, or customer service systems, providing efficient information delivery.
  • Gaming and Entertainment: Give game characters or virtual avatars personalized voices to enhance immersion.
  • Voice Cloning and Content Creation: Clone specific human voices for advertising dubbing, video production, or personalized voice content creation.