What is MoonCast
MoonCast is a zero-shotAI podcast generationproject that synthesizes natural podcast-style speech from plain text sources. Trained with long-context language models and large-scale speech data, it can generate podcast audio several minutes long, supporting Chinese and English. The naturalness and coherence of the generated speech maintain high quality in long audio generation. MoonCast uses specific LLM prompts to generate podcast scripts, which are converted into final podcast audio through a speech synthesis module. Users can quickly generate podcasts through simple commands and pre-trained weights.
MoonCast's main features
-
Long audio generation: It adopts an audio modeling method based on long-context language models, and based on large-scale long-context speech data, it can generate podcast audio several minutes long.
-
Enhanced naturalness: Through the podcast generation module, it generates scripts with natural details. These details are crucial for generating natural podcast speech. Experiments show that it significantly outperforms existing baseline models in naturalness, coherence, and other aspects.
-
Multilingual support: It supports Chinese and English podcast generation, using specific LLM prompts to generate podcast scripts.
-
Zero-shot speech synthesis: Based on a few seconds of reference audio, it can synthesize realistic speech and maintain good speech quality and speaker similarity when processing long audio.
MoonCast's technical principles
- Multi-stage training: MoonCast's training process is divided into three stages:
-
First stage: The model learns to generate short sentences and single-speaker speech, mastering zero-shot speech synthesis capabilities.
-
Second stage: The model processes long audio that is not colloquial, such as e-books, improving the stability of long-context generation.
-
Third stage: The model learns to generate long dialogue audio containing rich colloquial details, mastering complex podcast generation techniques.
-
- Short-segment-level autoregressive audio reconstruction: MoonCast innovatively adoptsshort-segment-level autoregressive audio reconstruction technology. It allows the model to stream-reconstruct the current short audio segment based on already reconstructed content, improving the coherence of audio reconstruction.
- Spontaneity enhancement: To enhance the spontaneity of podcasts, MoonCast uses a podcast generation module to generate scripts with spontaneous details. These details include filler words, response words, and random pauses, making the dialogue more natural and realistic.
MoonCast's project address
- Project official website：https://mooncastdemo.github.io/
- Github repository：https://github.com/jzq2000/MoonCast
- arXiv technical paper：https://arxiv.org/pdf/2503.14345
- Online experience Demo：https://huggingface.co/spaces/jzq11111/mooncast
MoonCast's application scenarios
- Content creation: MoonCast can transform various text content (such as stories, technical reports, news, etc.) into engaging podcast audio.
- Education field: In the education field, MoonCast can transform teaching materials (such as academic papers, e-books, etc.) into podcast form, helping students better understand and absorb knowledge.
- Entertainment industry: MoonCast can generate podcasts with a natural conversational style, suitable for creating entertainment content.
- Business applications: In the business field, MoonCast can be used to generate podcasts for internal corporate training materials, or to convert press releases, product introductions, and other content into audio form for marketing and customer communication.
- Personal Use: For individual users, MoonCast can help them convert their blogs, diaries, and other content into podcasts, making it convenient to listen to while driving, exercising, and in other scenarios.