What is Step-Audio-EditX
Step-Audio-EditX is the world's first LLM-level audio editing large model open-sourced by StepFun, featuring three-axis fine-grained iterative control over "emotion, speaking style, and paralinguistics." It can arbitrarily increase or decrease the intensity of emotions such as anger, happiness, and sadness; stack styles such as coquettishness, whispering, and elderly multiple times; and insert 10 categories of natural paralinguistic tokens such as breathing, laughter, and sighs like subtitles. It comes with zero-shot TTS, enabling voice cloning without the target person's speech, and can instantly switch dialects by adding tags such as "[Sichuan dialect]" and "[Cantonese]" before the text. The model is entirely based on large-gap synthetic data for SFT+PPO, and in the post-training stage it can achieve attribute disentanglement and iterative control without additional encoders or adapters.
Main Features of Step-Audio-EditX
-
Emotion Editing: Dozens of labels such as anger, happiness, sadness, excitement, fear, surprise, and disgust, with multiple iterations to enhance or weaken intensity.
-
Style Editing: More than ten speaking styles such as coquettishness, whispering, elderly, child, serious, generous, and exaggerated, supporting stacking and fine-tuning.
-
Paralinguistic Insertion: Precisely insert 10 categories of natural tokens such as breathing, laughter, sighs, surprise (oh/ah), confirmation (en), dissatisfaction (hnn), question (ei), and uhm.
-
Zero-Shot TTS: Voice cloning without the target person's speech, and directly switch dialects by adding tags such as "[Sichuan dialect]" and "[Cantonese]" before the text.
-
Iterative Control: The same speech can be edited repeatedly, attributes are disentangled without crosstalk, and the effect is enhanced step by step.
-
Open-Source and Lightweight: Provides an 8-bit quantized version that can run on a single card with 8 GB of VRAM, with 4×A800/H800 achieving the best audio quality; includes inference/training code, Gradio Demo, and HF Space.
Technical Principles of Step-Audio-EditX
-
Dual-Codebook Audio Tokenization: In parallel, a 16.7 Hz/1024-entry "linguistic codebook" and a 25 Hz/4096-entry "semantic codebook" are interleaved in a 2:3 slice ratio, uniformly converting any speech into discrete tokens, preserving emotion and prosody information, and providing a "speech vocabulary" for subsequent direct LLM manipulation.
-
3B Audio LLM: Warm-started with a text-pretrained 3B model, text tokens and dual-codebook audio tokens are concatenated in chat format as input, and only audio tokens are output; training data text:audio = 1:1, fully leveraging the existing text LLM ecosystem for post-training.
-
Large-Gap Synthetic Data Driven: No additional encoders or adapters are introduced; SFT+PPO is performed only with paired data of "same text, different attributes (emotion/style/paralinguistics)"; the large gaps force the model to learn attribute disentanglement, enabling iterative intensity increase/decrease and multi-attribute stacking.
-
Flow Matching + BigVGANv2 Decoding: The dual-codebook tokens output by the audio LLM generate a Mel spectrogram through a DiT-flow matching module, and then the waveform is restored by the BigVGANv2 vocoder; 200k hours of high-quality training ensures pronunciation accuracy and timbre similarity.
-
Unified Framework: The same "tokenization→LLM→decoding" pipeline simultaneously supports zero-shot TTS, emotion/style/paralinguistic editing, speaking rate adjustment, and denoising, without task-specific modules, significantly reducing system complexity and inference cost.
Project Address of Step-Audio-EditX
- Project Official Website：https://stepaudiollm.github.io/step-audio-editx/
- Github Repository：https://github.com/stepfun-ai/Step-Audio-EditX
- HuggingFace Model Library：https://huggingface.co/stepfun-ai/Step-Audio-EditX
- arXiv Technical Paper：https://arxiv.org/pdf/2511.03601
Application Scenarios of Step-Audio-EditX
-
Audio Content Upgrade: Audiobooks, podcasts, and news readings can add emotions or styles such as "happy/sad/whisper" with one click, quickly generating multiple versions of audio without re-recording, enhancing immersion.
-
Video and Advertising Voiceover: Short videos, animations, and commercials can clone character timbres with zero samples, then iteratively add styles such as "coy, exaggerated, serious," achieving low-cost, multi-character, multi-emotion automatic voiceover.
-
Games/Virtual Idols: NPCs, virtual streamers, and VTubers can first clone a timbre using a single reference speech, then insert paralinguistic elements such as laughter, breathing, and sighs in real time, creating more vivid and sustainably interactive character voices.
-
Intelligent Customer Service and Voice Assistants: Customer service bots can, based on existing TTS, transform "plain replies" into "enthusiastic/soothing" emotions through editing, improving user experience; at the same time, dialect tags are supported to meet regionalized service needs.
-
Education/Language Learning: Online courses and language apps use styles such as "elderly/child/whisper" to generate age-appropriate pronunciations, or instantly switch standard Mandarin to Cantonese or Sichuanese, helping students imitate along and reducing teachers' recording costs.
-
Meeting Records and Accessibility: For meeting recordings containing noise or excessively long pauses, first perform "noise reduction + silence trimming" editing, then increase speech rate or add emotion as needed, generating clear and easy-to-read meeting summary audio.