AuK - Tencent Hunyuan's Open-Source Foundational Model for Speech Generation and Editing

What is AuK

AuK is Tencent Hunyuan's open-source 1.5-billion-parameterspeech generationand editing foundational model, which adopts a flow-matching diffusion architecture rather than autoregression. The model can complete 16 speech tasks through a unified natural language instruction interface, including zero-shot/instruction-based TTS, speech content editing, acoustic adjustment, paralinguistic editing, and denoising and vocal separation. The model provides AuK-Flash, a fast version with 4-step inference, which is about 4.5 times faster.

AuK's Main Features

• Zero-shot TTS: Clone a voice with a few seconds of reference audio and synthesize new speech.
• Instruct TTS: Generate corresponding speech from a pure text description of voice characteristics, without reference audio.
• Speech Content Editing: Replace, insert, or delete words in a recording without changing the timbre or intonation.
• Lyrics Editing: Preserve the melody and singing characteristics while only rewriting the lyrics.
• Pitch/Speed/Volume Editing: Precisely adjust acoustic attributes in units of semitones, speed multipliers, and decibels.
• Emotion and Timbre Editing: Change emotional coloring without changing the words, or convert timbre while preserving content.
• Accent Editing and Denoising: Remove accents, denoise and dereverberate, and improve clarity.
• Paralinguistic Event Editing: Add or remove breathing sounds, laughter, and coughing, and convert between normal speech and whispering.
• Speech Separation: Split multi-speaker audio by speaking order, or extract a target speaker by content description.
• Music Separation: Extract vocals from songs and remove accompaniment.

AuK's Technical Principles

• Overall Paradigm: AuK uses a flow-matching diffusion architecture to generate audio latent representations, which is naturally suited to "editing" tasks: it uses the latent representation of the original audio as the starting point for denoising and only modifies the parts required by the instruction, thereby better preserving the original timbre, melody, and unedited content. This is also the technical basis for its ability to cover the family of editing tasks.
• Three-Component Pipeline Architecture:The system consists of three parts: a multimodal large model responsible for reading natural language instructions and reference audio and generating semantic conditions; an audio VAE that encodes 24kHz waveforms into a 50Hz acoustic latent space; and the core hybrid Transformer, which is stacked from 10 layers of dual-stream MMDiT plus 20 layers of single-stream DiT, with a model width of 1536 and a total of 1.5 billion parameters.
• Large-Scale Instruction Alignment Training:The training data contains about 3.03 billion "instruction-audio" pairs, amounting to about 1.95 million hours of effective supervision. The key is that all tasks are unified into the same format of "instruction + input audio -> output audio," enabling the model to generalize to 16 tasks through a single interface.
• Post-Training and Human Preference Optimization:After basic training, the model undergoes human preference optimization and reinforcement learning post-training (RL post-training) to improve the naturalness, instruction-following ability, and auditory quality of generated speech.

Follow WeChat and reply "open source" to join theAI Open-Source Project Communication Group

How to Use AuK

• Confirm the Hardware Environment: Prepare an NVIDIA GPU with about 24GB of VRAM (such as RTX 3090/4090).
• Clone the Code Repository: Clone from GitHub Tencent-Hunyuan/AuK Clone the repository locally.
• Install the dependency environment: Configure the Python environment as required by the repository and install the necessary dependency packages.
• Download the model weights: Download the AuK weights and the accompanying VAE files from Hugging Face or ModelScope.
• Download the semantic conditioning model: Separately download and configure the multimodal large model Qwen2.5-Omni-3B for parsing instructions and reference audio.
• Prepare input materials: Prepare the target text, reference audio, or the original audio file to be edited.
• Write natural language instructions: Write instructions according to the task requirements, such as describing voice characteristics, specifying the sentence to replace, or adjusting the pitch/speaking rate, etc.
• Run inference: Enter instructions and materials through the command line (CLI), execute generation or editing, and wait for the output.
• Listen and adjust: Check the resulting audio, and if unsatisfied, adjust the instruction wording, sampling steps, or CFG strength and regenerate.

AuK's core advantages

• Unified interface, full task coverage: Through natural language instructions, the model can handle 16 speech tasks such as TTS, editing, enhancement, and separation, without needing to deploy a separate model for each task.
• Editing capability is rare in the industry: Supports fine-grained editing such as changing words, inserting sentences, adjusting pitch, and changing emotion, while preserving the original timbre and intonation to the greatest extent.
• Instruct TTS requires no reference audio: Speech can be generated from a textual description of voice characteristics, lowering the barrier to use.
• Flow matching architecture is more faithful: It uses diffusion-style generation rather than token-by-token prediction, using the original audio as the starting point for denoising during editing, so unmodified parts suffer almost zero damage.

AuK's project addresses

• Project official website：https://auk-project.github.io/
• GitHub repository：https://github.com/Tencent-Hunyuan/AuK
• HuggingFace model library：https://huggingface.co/tencent/AuK
• arXiv technical paper：https://arxiv.org/pdf/2609.08936

Comparison of AuK with similar competitors

AuK Application Scenarios

• Film/TV and short drama post-production dialogue repair: When an actor misspeaks a line or the script is slightly adjusted, directly replace individual words or sentences in the recording without re-recording in the studio, preserving the original performance emotion.
• Podcast and interview production: Remove slips of the tongue and filler words, adjust speech rate and rhythm, remove background noise and reverb, and turn rough-cut material into a finished piece with one click.
• Audiobook and AI voiceover production: Use Instruct TTS to generate differentiated timbres according to character descriptions; the same narrator can also have emotion and intonation adjusted in post-production, reducing the cost of multi-character production.
• Music creation and derivative works: Rewrite cover lyrics while preserving the original melody and vocal characteristics, or extract vocals from songs as remix material.
• Language training and accessibility applications: Remove accents and adjust speech rate and volume for users with accents or language impairments, generating clearer and more intelligible speech content.