VoiceSculptor - An open-source voice timbre design model by Northwestern Polytechnical University in collaboration with Yutu Intelligence and others

• Natural language-controlled timbre generation: Users can describe desired timbre characteristics through natural language instructions, such as gender, age, speech rate, pitch, and emotion, enabling highly customized speech synthesis.
• Fine-grained attribute control: The model supports fine adjustment of multiple speech attributes, including gender, age, speech rate, pitch, volume, and emotional expression, meeting personalized needs.
• Retrieval-Augmented Generation (RAG): Through retrieval-augmented technology, the model can better understand and generate speech corresponding to out-of-domain natural language instructions, improving generalization ability and robustness.
• Voice cloning and speech synthesis: The generated audio can be used as prompt waveforms for CosyVoice2's voice cloning and downstream speech synthesis tasks, enabling efficient timbre transfer and synthesis.
• Role-playing and diverse speech generation: Supports generating corresponding speech styles based on different role descriptions, such as suspense novel narrators, news anchors, and fairy tale narrators, adapting to various application scenarios.

Technical Principles of VoiceSculptor

• Overall architecture: VoiceSculptor consists of a Voice Design module and a Voice Clone module. The Voice Design module generates timbre and speech attributes based on the LLaSA model, and reconstructs them into audio with the XCodec2 decoder; the Voice Clone module uses the generated audio as prompt waveforms to achieve voice cloning and speech synthesis through CosyVoice2.
• Voice Design module: The Voice Design module uses the LLaSA model as its base, and through joint training on natural language instructions, fine-grained attribute tokens, and target text, maps text into speech feature representations, which are then converted into audio waveforms by the XCodec2 decoder, achieving natural language instruction-controlled timbre generation.
• Retrieval-Augmented Generation (RAG): The model introduces retrieval-augmented technology, using Qwen3-Embedding-0.6B to vectorize and store natural language instructions in the Milvus database. During inference, vector retrieval is performed on input instructions to match similar instructions, enhancing the model's understanding and generation ability for complex instructions.
• Voice Clone module: The Voice Clone module is based on CosyVoice2 Implementation: the audio generated by the voice design module is used as the prompt waveform input, and through timbre cloning technology, speech similar to the prompt waveform is generated, completing the downstream speech synthesis task.
• Training Data and Strategy: The training data includes a large number of speech samples annotated with timbre attributes. By combining continual pretraining and supervised fine-tuning, the model performance is improved, ensuring the model's generalization ability and generation quality in different scenarios.

VoiceSculptor Project Address

• GitHub Repository：https://github.com/ASLP-lab/VoiceSculptor
• HuggingFace Model Library：https://huggingface.co/ASLP-lab/VoiceSculptor-VD

VoiceSculptor Application Scenarios

• Personalized Speech Synthesis: Provide users with customized voice services, for example, generating speech in a specific style based on user descriptions, for use in personal assistants, smart speakers, and other devices, meeting users' personalized needs for voice style.
• Virtual Vocals and Digital Humans: Generate natural and diverse speech for virtual streamers, virtual customer service agents, virtual characters, etc., improving the expressiveness and interactivity of virtual characters and enhancing the user experience.
• Audio Content Creation: In fields such as audiobooks, radio dramas, and animation dubbing, quickly generate speech in different styles based on text content, improving content creation efficiency and reducing production costs.
• Interactive AI: Provide natural-language-controlled voice output for interactive AI systems such as chatbots and intelligent customer service, enhancing the system's naturalness and user-friendliness.
• Education and Training: Generate vivid voice explanations for educational software, such as simulating dialogues between different characters, speeches by historical figures, etc., improving the fun and immersion of learning.