tts-voice-synthesis

TTS Speech Synthesis Service

Task Objective

• This Skill is used to: convert text into high-quality speech, supporting timbre cloning, emotion adaptation, streaming generation, and multilingual support
• Capabilities include:
  • Automatic character timbre collection and cloning (extract timbre features from reference audio)
  • Anthropomorphic semantic-adaptive dubbing (automatically adjust speech intonation, speed, and pitch according to text emotion)
  • Streaming real-time dubbing (supports generating speech while inputting text)
  • Multilingual and dialect support (Chinese, English, and multiple dialects)
  • Dual model selection (1.7B high-quality model, 0.6B fast model)
• Trigger conditions: use when text needs to be converted to speech, a specific timbre needs to be cloned, or emotional dubbing needs to be generated

Prerequisites

• Model download: download the corresponding weights according to the selected TTS model, see references/model_config.md
• Hardware requirements:
  • GPU: a GPU with 8GB+ VRAM is recommended (the 0.6B model can run on CPU)
  • Memory: 16GB+ system memory is recommended
  • Disk space: at least 10GB of available space (model weights are about 3-5GB)
• Dependency configuration: ensure the required Python dependency packages are installed

Operation Steps

Mode 1: Basic Speech Synthesis

1. Text preparation

   • Confirm the text content to be synthesized
   • The agent will analyze the text's emotional and semantic features

2. Select timbre

   • Use a preset timbre (see references/model_config.md)
   • Or use an already cloned custom timbre

3. Execute synthesis

   • Call scripts/tts_generate.py to perform speech generation
   • Automatically set speech parameters according to the emotion analysis results

4. Verify output

   • Check the quality of the generated audio and the degree of emotional matching
   • If needed, adjust parameters and regenerate

Mode 2: Timbre Cloning

1. Prepare reference audio

   • Provide a reference audio file of the target timbre (3-30 seconds, clear speech)
   • Ensure the reference audio has no background noise and clear sound quality

2. Extract timbre features

   • Call scripts/voice_clone.py to extract timbre features
   • Save as a reusable timbre model

3. Use the cloned timbre

   • Use the extracted timbre model to generate speech
   • Can be applied to dubbing for different texts

Mode 3: Streaming Real-Time Dubbing

1. Text segmentation

   • Process long text in segments (the agent completes this automatically)
   • Ensure the segmentation is natural and does not truncate semantics

2. Streaming generation

   • Call scripts/tts_generate.py to enable streaming mode
   • Gradually generate and output audio segments

3. Real-time merging

   • Merge the generated audio segments in real time
   • Output the complete dubbing file

Mode 4: Emotion-Adaptive Dubbing

1. Text Sentiment Analysis

   • The agent analyzes the sentiment tendency of the text (happy, sad, angry, calm, etc.)
   • Identify key emotional words and tone

2. Voice Parameter Adjustment

   • Automatically adjust based on emotion:
     • Speech rate (slow down when sad, speed up when excited)
     • Pitch (lower when sad, raise when excited)
     • Volume (adjust according to emotional intensity)

3. Generation Verification

   • Generate emotional speech
   • Verify whether the emotional expression is accurate

Resource Index

• Core Scripts:
  • scripts/tts_generate.py(TTS speech generation)
  • scripts/voice_clone.py(voice cloning)
• Reference Documentation:
  • references/model_config.md(model configuration and selection guide)
  • references/emotion_guide.md(emotion annotation and adaptation guide)
  • references/usage_examples.md(usage examples)

Notes

• Model Selection:
  • 1.7B model: higher audio quality, suitable for high-quality dubbing, audiobooks, and other scenarios
  • 0.6B model: faster speed, suitable for real-time interaction, intelligent customer service, and other scenarios
• Voice Cloning:
  • The reference audio should be clear and free of background noise
  • A duration of 5-15 seconds is recommended, with a minimum of no less than 3 seconds
  • Single-speaker audio works best; avoid mixed multi-speaker audio
• Streaming Generation:
  • Suitable for long texts and real-time interaction scenarios
  • Will produce multiple temporary audio segments
• Emotion Adaptation:
  • Text sentiment analysis is performed by the agent
  • The final effect depends on the accuracy of the emotion annotation
  • Voice parameters can be manually adjusted for fine-tuning

Usage Examples

• Basic Speech Synthesis:

  python scripts/tts_generate.py \
    --text "你好，欢迎使用语音合成服务" \
    --output_path ./output/hello.wav \
    --model_size 1.7B \
    --voice default
  

• Voice Cloning:

  python scripts/voice_clone.py \
    --reference_audio ./reference.wav \
    --voice_name my_voice \
    --output_dir ./voices
  

• Emotional Dubbing:

  python scripts/tts_generate.py \
    --text "今天真是太开心了！" \
    --output_path ./output/happy.wav \
    --emotion happy \
    --speed 1.2 \
    --pitch 1.1
  

• Streaming Generation:

  python scripts/tts_generate.py \
    --text_file ./long_text.txt \
    --output_path ./output/stream_output.wav \
    --streaming true