What is PodAgent
PodAgent is a podcast generation framework jointly launched by The Chinese University of Hong Kong, Microsoft, and Xiaohongshu. Based on simulating real talk show scenarios, it uses a multi-agent collaboration system (including hosts, guests, and scriptwriters) to automatically generate rich and structured dialogue content. PodAgent has built a diverse voice library for accurately matching roles with voices, ensuring the naturalness and immersion of the audio. PodAgent introduces large language model (LLM)-basedspeech synthesistechnology to generate expressive and emotional speech, making podcasts more engaging. PodAgent has launched comprehensive evaluation metrics to measure the quality of generated podcasts, ensuring the professionalism and diversity of the content.
Main Features of PodAgent
- Generate high-quality dialogue content: Automatically generate rich and diverse dialogue scripts covering various topics.
- Voice-role matching: Dynamically match the most suitable voice based on the character's personality and content context.
- Speech synthesis and expressiveness enhancement: Adjust the tone, rhythm, and emotion of the speech according to the emotion and context of the dialogue content, making the podcast more vivid.
- Generate a complete podcast structure: Support adding appropriate sound effects and background music to generate a complete podcast structure. Support multilingual generation to adapt to the needs of different scenarios and audiences.
- Evaluation and optimization: Provide comprehensive evaluation metrics to measure the quality of generated podcasts, including the richness of dialogue content, the accuracy of voice matching, and the expressiveness of speech.
Technical Principles of PodAgent
- Multi-agent collaboration system：
- Host: Responsible for formulating the dialogue outline and guiding topic discussions.
- Guest: Provides professional insights and opinions according to the role setting.
- Scriptwriter: Integrates dialogue content and optimizes the coherence and diversity of the script.
- Voice feature analysis and matching: Build a voice library, analyze voice features (such as timbre, intonation, emotion, etc.), and match the most suitable voice for each role. Use open-source datasets (such as LibriTTS and AISHELL-3) to extract voice samples, and generate a diverse voice library based on deduplication and filtering.
- LLM-guided speech synthesis: Use large language model (LLM)-based speech synthesis technology to convert text content into natural and expressive speech. Use the speaking style predicted by the LLM as instructions to guide the speech synthesis model (such as CosyVoice) to generate speech that matches the emotion of the content.
- Comprehensive evaluation metrics: Launch a set of evaluation metrics for measuring the quality of generated podcasts. The metrics include the lexical diversity, semantic richness, and information density of dialogue content, as well as the accuracy of voice matching and the expressiveness of speech. Use LLMs as evaluation tools to compare and score generated content.
PodAgent Project Address
- GitHub Repository：https://github.com/yujxx/PodAgent
- arXiv Technical Paper：https://arxiv.org/pdf/2503.00455
Application Scenarios of PodAgent
- Media and content creation: Quickly generate high-quality podcast programs covering topics such as news, culture, and technology, saving creation time and cost.
- Education and learning: Generate educational podcasts, such as language learning and academic lectures, providing vivid and interesting learning experiences.
- Corporate promotionPromotion: Create branded promotional podcasts, share product stories or industry insights, and enhance brand influence.
- Self-media and personal branding: Help creators quickly generate podcast content, break through creative bottlenecks, and increase content appeal.
- Entertainment and creativity: Generate entertainment podcasts such as fictional stories and comedy talk shows, providing an immersive auditory experience.