You may also be troubled by this: when using AI to generate a blockbuster, the visual quality is already spot on, but the moment a character opens their mouth, it instantly breaks the immersion.
On screen, they are clearly going through a life-or-death moment, yet the voiceover sounds like they are calmly reading a product manual; if there is finally some emotional fluctuation, it all relies on stiff shouting... What is even worse is that the character's timbre is completely different across different clips, making it very hard to maintain timbre consistency.
Not to mention that afterward you still need to add ambient sound, sound effects, background music, lip-sync... very troublesome.
Not long ago, at the Volcano Engine FORCE Original Power Conference, ByteDance officially releasedDoubao Audio Generation Model 1.0, and rich, emotional voice materials can also be generated end-to-end.
We input a prompt, and Doubao Audio Generation Model 1.0 can generate vocals, sound effects, score, and scene sound all in one package. It not only saves the cumbersome multi-track editing, but can also simulate the subtle breathing and emotional changes when real people speak, making AI voices sound more natural and more human。
How does it perform in real creation? Today let's test it together and see.
01. AI Voice Magic in Multiple Scenarios
We open the Volcano Ark Experience Center, select Doubao Audio Generation Model 1.0. Ordinary users have a 30-minute free trial quota, and later can call it through the API.
We input the effect prompt and synthesis text, click generate, and we can get a complete audio clip containing vocals and ambient sound.
Single-person voiceover
I tried generating a monologue for a novel character.
Background music is laid lightly underneath, mainly low strings, distant drums, and ethereal female vocal humming. The beginning is oppressive and solemn, like the silence before wind and snow; as the character's emotions advance, the music gradually intensifies, but do not let it overpower the vocals. The overall atmosphere goes from going alone into the setup to breaking the setup and establishing the way, solemn, tragic, restrained, and inspiring. Xie Chang'an (young female voice, cool and transparent timbre, stable vocal line, slightly oppressive, gradually firm and inspiring in the later section) calm, restrained, as if establishing her own way under the gaze of the crowd, says: "All the nobles in the court cherish their own lives, so it can only be someone as small as me who takes action. My way is the way of all living beings. All living beings, everyone can walk it. If there is a setup, there must be a way to break it. Rather than going along with it, it is better to fight for survival in a desperate situation. Perhaps a way out will appear, right on the unexpected third path."
At first I wrote young female voice, cool and transparent, but the result was rather soft. It had the fairy-like quality, but not enough pressure. Later I changed the prompt to young female mezzo-soprano, not sweet, not soft and cute, not girlish, and the effect was very close to a strong female lead.
The simultaneously generated background music is also very imposing, and fits the character's voice and emotions very well.
Multi-role voiceover
We upload a two-person crosstalk routine and set up 2 roles with fairly big differences:
Background music is extremely light, mainly small-theater opening gongs and drums and short sanxian accents. At the beginning there is slight audience ambient sound. The overall atmosphere is lively, relaxed, and down-to-earth. Laughter can appear a small amount, but not frequently, and must not cover the dialogue. Vocals must be clear and upfront.
Female comic lead (young female voice, bright and crisp, fast speaking speed, sharp-tongued, with a bit of Beijing flavor and playfulness, emotionally outgoing but not shrill) excited and smug, as if she has discovered a new tool, says: "Let me tell you, AI voiceover is really amazing now. I input the script, and it directly speaks it out for me."
Male捧哏 (middle-aged male voice, deep and full, speaking half a beat slower, steady tone, with a bit of dry humor and skepticism) calmly and skeptically responds: "What's so new about that? It could do it before too."
Female逗哏 (young female voice, raised tone, exaggerated but cute) says: "Was that called speaking before? Before, that was called elevator announcements."
Male捧哏 (middle-aged male voice, half a beat slower, seriously picking up the joke) says: "Pretty disciplined, actually."
Female逗哏 (young female voice, quickly catching it, with a laugh) says: "Discipline yes, but not a bit of emotion."
Male捧哏 (middle-aged male voice, chuckles softly) says: "The main thing is equality for all beings."
Female逗哏 (young female voice, continuing excitedly, speaking quickly) says: "Now it's different. If you have it tell a children's story, it can be gentle; if you have it perform a suspense short drama, it can lower its voice; if you have it do a strong female lead, it can even carry a bit of the drive to break through."
Male捧哏 (middle-aged male voice, deliberately skeptical) says: "Then what if you have it perform crosstalk?"
Female逗哏 (young female voice, pauses for a moment, seriously) says: "Isn't that exactly what we're doing right now?"
Male捧哏 (middle-aged male voice, half a beat slower, suddenly realizing) says: "So I was generated too?"
After testing the two-person dialogue, the naturalness of the two-person conversation is much better than ordinary TTS.
The female逗哏's rhythm is faster, and the emotional transitions are very natural; the male捧哏 reacts more slowly, each timbre is very distinctive, and timbre consistency is also maintained very well.
The key point is that the Doubao audio generation model 1.0 also directly generated the audience laughter from a live crosstalk performance, very naturally.
With one sentence, you can feel an immersive scene, and the efficiency improvement of AI dubbing is evident.
Audiobook long text
Complex audiobooks often require the coordination of multiple characters and multiple environmental sounds. We tried a complex ancient-style suspense ensemble:
Background music is lightly laid underneath, mainly low strings, distant drums, and a cold-toned guqin. The overall atmosphere is grim, cold, and oppressive, with a sense of ancient-style political intrigue. In the first chapter, the palace gate and court scenes, the music is solemn and tense, like wind and snow bearing down on a city; in the second chapter, the side-hall private discussion scene, the music is lower and darker, adding a slight sense of suspense. Ambient sounds include wind and snow, the palace gate opening, the rustle of clothing, the crackle of lamp wicks in the hall, and the footsteps of imperial guards in the distance. The human voice must be clear and forward; music and ambient sound must not overpower the dialogue. Narrator (adult female voice, low and steady, strongly narrative, medium-slow pace, with vivid imagery and suspense, not a broadcast-announcer tone) is calm and restrained, as if telling of a court killing trap on a snowy night. Shen Zhaoxue (young female mezzo-soprano, cold, stable, low voice, supported by the chest, clear articulation, clean endings, not sweet, not soft and cute, not girlish) is enduring, calm, and sharp; in the early stage she suppresses anger, and later gradually reveals the resolve and sense of control of breaking the situation. Xiao Cheng (young male voice, low and clear-cool, not fast, restrained, with the distant and probing feel of the Eastern Palace crown prince) is cautious, steady, and repressed, like someone who has lain low for years testing a knife that may wound him. Pei Jingzhi (middle-aged/older male voice, low and thinly cold, slow pace, steady articulation, with the oppressive and scrutinizing feel of a powerful minister) is composed, arrogant, and dangerous, like someone accustomed to controlling the court encountering an uncontrollable variable for the first time. The little emperor (boy's voice, somewhat young but trying hard to be proper, with nervousness and unease) is pressed down by the court situation, wanting to ask the truth yet afraid of it. Vice Minister Zhou (middle-aged male voice, slightly weak, pace steady at first then chaotic) is guilty, panicked, and forcing himself to stay calm. Imperial guards/defending general (adult male voice, low and clipped, obedient and tense) is solemn and on guard. Little eunuch (boy's voice, trembling, unsteady breath) is terrified, collapsing, and desperate to survive. On the day Shen Zhaoxue entered the capital, the obituary from the northern border arrived before she did. The obituary stated clearly: Shen Zhaoxue, grain route commissioner of the Zhenbei Army, encountered bandits while escorting military grain, and both person and cart fell into the Black Gorge, leaving no remains. But at dusk, she wore a fox fur cloak washed so pale it was almost white, led a thin horse, and stood outside the Vermilion Bird Gate. When the imperial guards at the gate saw the half-cracked bronze sparrow tally at her waist, their expressions changed on the spot. The bronze sparrow tally was the troop-transfer token bestowed by the late emperor to the Zhenbei Army, half in the northern border and half on the imperial desk. Everyone in the world knew that the northern half had disappeared ten years ago after the entire Shen family was imprisoned. Shen Zhaoxue raised her hand and placed the bronze sparrow tally in the defending general's palm. "Please announce me," she said. "A dead person has returned to the capital and wishes to see the living lords." Wind and snow poured into the palace gate, and the defending general's hand trembled. Half an hour later, the Taiji Hall was fully lit. The hall was packed with people. Left Chancellor Pei Jingzhi wore a purple robe, an ivory tablet pressed within his sleeve. He was over fifty, with very thin eyelids; when he raised his eyes to look at people, it always seemed as if he were looking at a page about to be burned. Crown Prince XiaoCheng sat below the imperial steps, his fingertips slowly rubbing a teacup. Beside him, the little emperor was only twelve years old, and the dragon robe on his shoulders was as loose as if borrowed. Shen Zhaoxue knelt in the hall, snowmelt dripping from the hem of her clothes onto the blue bricks. Pei Jingzhi spoke first. "Shen, daughter of criminals, since you did not die, why not first go to the Ministry of Justice and turn yourself in?" Shen Zhaoxue raised her head. Her face was very pale, but her gaze was steady. "If your servant girl went first to the Ministry of Justice, then tonight all you lords would not hear the news from the northern border." Someone in the hall sneered. "You, a descendant of a criminal official, what news could you bring?" Shen Zhaoxue took a roll of oilcloth from her sleeve and presented it with both hands. "One hundred seventy thousand shi of military grain left Luocang three months ago, and the accounts say it has entered the northern border. But the Zhenbei Army received only fifty thousand shi." The hall fell silent. Pei Jingzhi did not move. But the crown prince Xiao Cheng gently set down his teacup. "Continue." Shen Zhaoxue said, "The missing one hundred twenty thousand shi, converted into silver, would be enough to support thirty thousand private soldiers for a year." Someone denounced her on the spot: "Outrageous! Do you know what you are saying?" "I know." Shen Zhaoxue looked at that person. "Lord Zhou, Right Vice Minister of Revenue, the ink paste on the warehouse release document you approved had cinnabar mixed into it. On the half-length grain tally your servant girl picked up from Black Gorge, there is exactly this seal as well." The blood drained completely from Vice Minister Zhou's face. Pei Jingzhi finally raised his eyes. "Miss Shen did not die after falling off the cliff, but she has grown a sharp tongue." Shen Zhaoxue smiled. "Before falling off the cliff, your servant girl did not like to talk either." Outside the hall, the wind grew heavier. The little emperor clenched the armrest of the dragon throne and asked softly, "Then what about the grain?" When these words fell, all the officials in the hall lowered their heads. Shen Zhaoxue looked at that young emperor. "The grain is gone." She paused. "The northern border will soon be gone too." The crown prince Xiao Cheng's gaze sank. "How is the Zhenbei Army?" "Seven days ago, the Qiangrong broke through Frost River Mouth. The Zhenbei Army retreated to guard Chensha City, and only two days of grain remain in the city." The little emperor stood up. "Why has no one reported this?" Shen Zhaoxue did not answer immediately. She took out a second item from her bosom. A broken arrow. A half-length strip of red cloth was wrapped around the arrow shaft, and the red cloth had been soaked black by blood. "Because the person delivering the report died thirty li before entering the capital." She placed the broken arrow on the ground. "This is the sixth." No one in the hall laughed anymore. The crown prince Xiao Cheng slowly rose and walked down the imperial steps. He stopped three paces from Shen Zhaoxue, his gaze falling on the unmelted snow on her shoulder. "What do you want?" "Open the granaries." "Only open the granaries?" "Also a squad of imperial guards to escort me to Luocang to collect the grain." Pei Jingzhi finally laughed. "You want troops?" Shen Zhaoxue looked at him. "Prime Minister Pei is wrong. What I want is the road." Pei Jingzhi's smile faded. "Luocang is in the capital region, and the garrison troops guarding the granary are all under the coordination of the Ministry of Revenue. A daughter of a criminal official, by what right would she open the granary?" Shen Zhaoxue reached into her sleeve. The imperial guards all pressed their sabers in unison. What she took out was a blood letter. Most of the writing on the blood letter was blurred, and only the last line was still legible. Your subject Shen Huaishan is willing to exchange the lives of his whole household for three years without war in the northern border. Shen Huaishan was her father. Ten years ago, he was convicted of secretly colluding with the Qiangrong, and his whole household was imprisoned. Shen Zhaoxue was fifteen that year. She knelt at the gate of the Ministry of Justice for three days, and no one dared give her a sip of water. Now that blood letter, which had never been delivered to the imperial presence, lay in the hall like a bone that had arrived too late. The little emperor's face turned pale. Pei Jingzhi's fingers moved within his sleeve. Shen Zhaoxue saw it. She bowed and knocked her head on the ground, her voice not loud, yet it overpowered the wind and snow outside the hall. "Your servant girl relies on a memorial from the Shen family that could not be delivered ten years ago, on the lives of the seventy thousand soldiers of the northern border, and on the people in Chensha City who have not yet breathed their last." She raised her head. "If it is still not enough, your servant girl is willing to submit a military pledge." Xiao Cheng asked, "How many days?" "Three days." "If the grain does not reach Chensha City?" Shen Zhaoxue looked at him and said, word by word, "I will die at the city gate." The hall was so quiet that the crackling of the lamp wick could be heard.
Doubao Audio Generation Model 1.0 automatically recognizes audiobook content, such as the description of wind and snow pouring into the palace gate in the text, and automatically infers and matches suitable sound effects.
The female lead's voice is calm and restrained, the minister's voice is slow and carries a sense of pressure, and the narration and the different character voices all have high recognizability.
The volume ratio among vocals, ambient sounds, and background music is also relatively moderate, saving us the cumbersome steps of repeatedly dragging volume sliders in editing software.
However, Doubao Audio Generation Model 1.0 can generate at most 2 minutes of audio at a time. If you want to make a complete audiobook, you need to generate it in segments.
The generation effect for long texts is average, the order of some dialogues may be reversed, and the recognition of polyphonic characters is not very stable, so pronunciation annotations are needed.
AI short drama dubbing
Let's try a slice-of-life AI short drama again. Ordinary TTS can only read lines, but short dramas require the voice to have a sense of space.
Background music plays lightly underneath, mainly warm piano, soft strings, and faint city ambient sounds. The overall atmosphere is realistic and slice-of-life, with a touch of warmth and a twist. Do not make it suspenseful or horror. Ambient sounds include faint chatter in a coffee shop, clinking cups, a doorbell, a phone vibrating, and vehicles on the street after rain. The voices must be clear and forward; the music must not overpower the dialogue. Narrator (adult female voice, gentle and steady timbre, medium pace, with a slice-of-life storytelling feel): calm and delicate, as if telling a small thing that happened to ordinary people. Lin Xia (young female voice, clean and bright, tone a little tired but very restrained): from feeling lost and forcing herself to keep up appearances, to gradually letting go in the latter half. Zhou Yan (young male voice, low and gentle timbre, not fast-paced, sincere but a little clumsy): careful, guilty, trying to explain; do not use a domineering-CEO tone. Clerk (young female voice, light and natural, polite): brief appearance, slice-of-life. Chapter content: "The Seat by the Window" Narrator: "Lin Xia and Zhou Yan agreed to meet at that coffee shop, on the seventh day after their breakup." Narrator: "The rain had just stopped, and the leaves outside the window were still dripping. Lin Xia sat in the seat by the window, with two cups of coffee on the table. One was hot, and one had already gone cold." Clerk: "Hello, would you like me to get you a fresh hot one?" Lin Xia: "No, thanks." Narrator: "After she said that, she glanced at her phone. Zhou Yan was twenty-six minutes late." Narrator: "When the wind chime at the door rang, Lin Xia had already rehearsed the line 'Don't contact me again' three times in her head." Zhou Yan: "Sorry, I'm late." Lin Xia: "You've always been good at being late." Zhou Yan: "There was really a traffic jam today." Lin Xia: "Last time it was overtime, and the time before that it was a last-minute meeting. Zhou Yan, I didn't come here to listen to reasons." Narrator: "Zhou Yan stood by the table, holding a paper bag. The opening of the bag had gotten a little wet from the rain." Zhou Yan: "I know." Lin Xia: "Then sit down and finish what you have to say." Narrator: "He sat down across from her, but did not touch the coffee that had already gone cold." Zhou Yan: "You said that day that I had never put you first." Lin Xia: "Isn't that true?" Zhou Yan: "It is." Narrator: "Lin Xia looked up at him. The answer was too straightforward, which instead made the accusation she had prepared catch in her throat." Zhou Yan: "I always thought that if I did well at work first, saved enough for the mortgage first, and stabilized life first, we would be a little better." Lin Xia: "But what I got was you being absent again and again." Zhou Yan: "So today I'm not here to beg for your forgiveness." Lin Xia: "Then why did you come?" Narrator: "Zhou Yan pushed the paper bag in front of her." Zhou Yan: "To return your things." Narrator: "Lin Xia opened the paper bag. Inside was not the scarf she had left at his place, nor the keys." Narrator: "It was a stack of bus tickets, movie ticket stubs, and more than a dozen takeout receipts." Lin Xia: "What is this?" Zhou Yan: "You say I don't remember anything. Actually, I do remember, I just didn't say it." Narration: "Lin Xia flips to the very bottom and sees a faded sticky note." Narration: "On it are words she wrote two years ago: If we ever argue in the future, go to the seat by the window and make up." Lin Xia says nothing. Zhou Yan: "I know it's a bit late to say this now." Lin Xia: "It is late." Zhou Yan: "Yeah." Narration: "A car passes outside the window, and the sound of water lightly splashes up." Zhou Yan: "But I want to give them back to you. Not to make you turn back, but to tell you that I haven't forgotten those days." Lin Xia: "Then why didn't you say it earlier?" Zhou Yan: "Because I always thought doing was more important than saying." Lin Xia: "And later?" Zhou Yan: "Later I realized that only doing without saying also makes people feel they don't matter." Narration: "Lin Xia looks down at that sticky note. The corner of the paper has curled up, but the words are still clear." Lin Xia: "Zhou Yan, I don't want to wait anymore for someone who is always late." Zhou Yan: "I know." Lin Xia: "But this cup of coffee, I can finish with you." Narration: "Zhou Yan is stunned for a moment, then slowly smiles." Zhou Yan: "It's already cold." Lin Xia: "Then let's get a hot one." Narration: "The server comes over and takes away the cold coffee. Outside the window the clouds scatter a little, and sunlight falls on the seat by the window." Ending sound effect: the cup is gently set down, the doorbell rings once, and the background music softly comes to a close.
The characters' dialogue is very natural and lets you feel the flow of emotions. The sound of rain and the flipping of cards both help us build the scene.
Sound is no longer just an accessory added at the very end after the video is finished; it can participate in creation starting from the script stage.
Voice Replication
Doubao Audio Generation Model 1.0 currently generates at most 2 minutes of audio in a single pass. If we want to create long audio, or make a sequel, how do we make sure the voices don't break character?
We can upload reference audio, or use previously generated audio as reference audio, supporting up to 3 clips at a time, and specify in the prompt that a certain character uses a certain timbre.
For example, let's try replicating Doubao's voice:
The music begins with light jazzy drums, a short bass, and a few playful piano notes, with background sounds of a small theater audience chatting softly, cups clinking, and scattered laughter. The overall atmosphere is relaxed, lively, and feels like an urban nightclub stand-up comedy scene. After the performer starts speaking, the music quickly lowers, keeping only a very light bass groove. Audience laughter, catcalls, and applause can appear naturally, but must not overpower the vocals.
Stand-up comedian (young female voice, Mandarin, low pitch, slightly hoarse voice, medium-fast speaking rate, strong complaining rhythm, with natural pauses and punchline pauses, no broadcaster tone, performer is) says in a relaxed, self-mocking state, as if chatting with the audience in a small theater: "I recently discovered that the biggest impact of AI isn't replacing jobs, it's finally making my mom confirm that I really am useless."
The audience chuckles softly.
Stand-up comedian (setting it up seriously) continues: "Before, when my mom ran into a problem she'd call me. Phone broken, call me; TV has no sound, call me; can't find the WeChat group, call me. Now it's different. She asks AI first."
Pause for half a second.
Stand-up comedian (voice suddenly a bit lower) says: "After asking AI, then she calls me."
The audience laughs.
The stand-up comedian (helplessly) says: "She said, AI gave her an answer, but she wasn't reassured and wanted me to confirm it. I said, Mom, you've demoted me from technical support to manual review." The audience laughs. The stand-up comedian (speaking faster) says: "The scariest part is, she now uses AI to write WeChat Moments posts. Before, her Moments were: Today I made dumplings. Now it's: Time settles in the flour, family love glows in the folds." Pause. The stand-up comedian (lowering voice) says: "My dad read it and asked her, are these dumplings for eating today, or for exhibition?" The audience laughs loudly. The stand-up comedian (continuing to roast) says: "My mom even seriously asked me, will posting this be too ordinary? I said no, it's pretty good, it just doesn't sound like you. She asked where it doesn't sound like her. I said when you usually post on Moments, you don't even use punctuation, and suddenly family love glows in the folds—relatives will think you've been possessed by flour." The audience laughs.
The generated timbre has a high similarity to the reference timbre, and it retains the self-deprecation and relaxed feel needed for stand-up comedy. The pauses at punchlines and the interspersed audience laughter are very natural.
Doubao Audio Generation Model 1.0 can not only clone timbre, but also bring in more emotions, more like using timbre to complete a performance.
02. Some Sharing
With AI voiceover in the past, we just fed text to it; now, in the prompt, we need to explain the role like a director—clearly write the character's age, timbre characteristics, current emotions, action breaths, and what sounds should be in the background. The more concrete the details given, the closer the generated effect is often to expectations.
The previously cumbersome workflow of voiceover, music, finding sound effects, and aligning the mix can now quickly produce a complete first draft through a reasonable Prompt, and the efficiency improvement is obvious. The production speed of short dramas, advertisements, courses, and virtual IPs will clearly become faster.
Currently, the Volcano Ark Experience Center has opened experience for Doubao Audio Generation Model 1.0, and ordinary users can get 30 minutes of free experience quota. In the future, it will also be integrated into everyday tools such as CapCut and Tomato Novel, and the barrier for ordinary people to create audio content will continue to drop.
If in the past AI voiceover solved whether there is sound, then Doubao Speech Model 1.0 is beginning to solve whether the sound has drama.
Of course, as version 1.0, the Doubao Audio Generation Model 1.0 still has room for refinement and optimization in some relatively complex physical sound field changes, polyphony, and accent details. But the end-to-end generation potential demonstrated by the Doubao Speech Model 1.0 has already let us see the prototype of a transformation in audio productivity.
As the AI toolchain for images, videos, copy, and audio becomes increasingly complete, AI dubbing will also become a key link in improving the content experience.
Original link:Sound has gone from "audible" to "dramatic"; AI dubbing has really improved this time