SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI audioConfucius4-TTS

Confucius4-TTS

Confucius4-TTS is an open-source multilingual and cross-lingual zero-shot text-to-speech (TTS) system from NetEase Youdao, based on a "speech encoder + large language model (LLM)" architecture. While synthesizing high-quality speech, it maintains the consistency of the speaker's timbre across languages. It supports 14 languages in total—Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese—and can switch seamlessly between languages. The project offers capabilities such as unconstrained voice cloning (no reference text transcription required), cross-lingual timbre transfer (synthesizing accent-free speech), zero-shot timbre transfer (cloning a timbre without additional training), and emotion transfer (cloning tone and emotion), and demonstrates strong generalization in real multilingual scenarios. Usage methods include example scripts, a Python API, a vLLM accelerated backend, a Gradio web interface, and a FastAPI server, supporting both non-streaming and streaming generation. The code and models are hosted on GitHub, and an online demo is also provided.

Open website

About this tool

Confucius4-TTS is an advanced LLM-based text-to-speech (TTS) system open-sourced by NetEase Youdao, focused on multilingual and cross-lingual speech synthesis. The project slogan is "One voice. Any language."

Purpose

The system is mainly used for multilingual and cross-lingual speech synthesis. It can generate speech across languages while maintaining consistency of the speaker's timbre identity, making it suitable for scenarios that require reusing the same timbre across different languages. An official online demo address is provided:https://confucius4-tts.youdao.com/gradio .

Main Capabilities

  • Supports 14 languages: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese, and states that more languages will be supported later.
  • Unconstrained Voice Cloning: no reference text transcription is required.
  • Cross-Lingual Voice Transfer: synthesizes accent-free speech across 14 languages.
  • Zero-Shot Voice Transfer: clones a voice without additional training.
  • Seamless Emotion Transfer: not only clones the timbre, but also clones the emotion.
  • Robust Generalization: stable performance in real multilingual scenarios.

Usage

  1. Basic running process:
    • Clone the code repository.
    • Create and activate a conda environment.
    • Install dependencies.
    • If access to HuggingFace is restricted in your environment, you can set a mirror endpoint before running.
    • Use the provided example.py script for zero-shot TTS synthesis.
    • You can also use the Python API directly.
  2. Accelerated backend:
    • The basic path uses HuggingFace Transformers to run the Text2Semantic (T2S) autoregressive stage.
    • If you want to speed up T2S generation, you can switch to the vLLM backend (example_vllm.py) to use PagedAttention to accelerate the LLM stage.
    • Currently supports vLLM 0.16.0 (V1 engine). Older versions have not been tested and may not run, because the model registration and GPUModelRunner monkey-patch target the internal implementation of the v1 engine.
    • Note: vLLM has specific requirements for GPU architecture, drivers, and CUDA, and may not run on your hardware, and installing it into the base environment may break other packages. It is recommended to clone a separate conda environment from the base environment, and then install only the additional vLLM dependencies.
  3. Streaming generation:
    • Through the --stream flag, both non-streaming and streaming are supported.Generate.
  4. Gradio web interface:
    • Provides a Gradio Web UI for interactive zero-shot voice cloning in the browser. Upload reference audio, enter text, select a language, and click Generate.
    • After startup, you can open http://<server-ip>:7860 in the browser. The reference audio is uploaded to the server via HTTP, so the client browser does not need to share a filesystem with the server.
  5. FastAPI server:
    • If programmatic access is needed (for example, calling TTS from another service/machine), the FastAPI server can be used. It exposes both non-streaming and streaming endpoints, and accepts reference audio as an HTTP file upload, so the client and server do not need to be on the same host.
    • Request form fields: text, lang (for example zh, en, ja), reference (audio file upload).

Training pipeline

Confucius4-TTS adopts a "speech encoder + LLM" architecture, and the training pipeline covers two modules:

  • Text2Semantic (T2S): generates a sequence of semantic tokens based on text and speaker conditions.
  • Semantic2Acoustic (S2A): a flow-matching model that converts semantic tokens into mel spectrograms.

Training steps:

  1. Prepare pretrained models: download two external models; after downloading, the directory structure must meet the requirements.
  2. Prepare training data: training data is provided as a TSV file (tab-separated, no header) containing 5 columns.
  3. Configure the training/validation paths in config/train_t2s.yaml.
  4. Start T2S training: set the pretrained T2S checkpoint path in config/train_t2s.yaml, then train on a single node.
  5. Start S2A training: set the checkpoint path in config/train_s2a.yaml. t2s_checkpoint points to the frozen T2S backbone; s2a_checkpoint is optional and is used to resume from a pretrained S2A model. During S2A training, the T2S model, speaker encoder (Wav2Vec2-BERT), and style encoder (CAMPPlus) are all frozen, and only the flow-matching S2A model is trained.

Evaluation and applicable scenarios

Confucius4-TTS has achieved competitive results on multilingual and cross-lingual zero-shot TTS benchmarks, with strong intelligibility and speaker similarity across multiple languages. In the evaluation metrics, lower WER/CER is better (↓), and higher SIM is better (↑). Evaluations include CV3-eval cross-lingual, X-Voice Benchmark, MiniMax-MLS-Test, and others.

Limitations and dependencies

  • Currently supports 14 languages; more languages are said to be coming soon.
  • The vLLM backend has hardware and driver requirements, may not run in some environments, and may affect other packages in the base environment; using a separate conda environment is recommended.
  • Only vLLM 0.16.0 (V1 engine) has been tested; older versions may not work.

Acknowledgements

Confucius4-TTS is built on the following open-source projects: Qwen3-TTS (speaker encoder ECAPA-TDNN and text embedding projector architecture), CosyVoice (text normalization pipeline), Amphion / MaskGCT (semantic codec implementation), w2v-BERT 2.0 (semantic feature extraction and speaker conditioning), Seed-VC (flow-matching architecture reference), BigVGAN (high-fidelity neural vocoder for mel-spectrogram-to-waveform synthesis). If Confucius4-TTS is used in research or a project, the official recommendation is to cite the project.

Related tools

Browse category