Khala - An Open-Source AI Music Model by the Central Conservatory of Music and Tsinghua University

What is Khala

Khala is an open-source AI musiclarge model jointly launched by the Central Conservatory of Music and Tsinghua University, aimed at high-fidelity song generation. The model adopts a 64-layer deep acoustic token hierarchy, supports generating complete songs based on text descriptions and lyrics conditions, and reaches the first tier of open-source models in terms of vocal clarity, instrument separation, and lyric-rhythm alignment. The project has released its code, model weights, and deployment instructions. Running it locally requires 24GB of VRAM, and it is currently more suitable for technical users as an open-source foundation.

Khala's Main Features

• Text-to-Song Generation: Input a text description to generate a complete song containing vocals and accompaniment.
• Lyrics-Conditioned Generation: Supports melody and vocal generation based on custom lyrics, achieving synchronization between lyrics and music.
• High-Fidelity Audio Output: The generated audio is closer to finished quality in transients, overtones, and instrument textures.
• Separated Vocal and Accompaniment Generation: Through acoustic token hierarchy modeling, it reduces the problem of vocals and instruments blending into a blur.
• Time Alignment Control: Strengthens the temporal alignment between lyrics and sound features, reducing swallowed words, reversed words, and rhythm misalignment.

Khala's Technical Principles

• Acoustic Token Language Model: Khala is based on an acoustic token language model. The model breaks sound down into extremely fine discrete acoustic units, allowing the model to learn the organizational and sequential relationships between units.
• 64-Layer Deep Hierarchy: The model adopts a 64-layer deep acoustic token hierarchy, equivalent to microscope-level decomposition of sound. The fine-grained representation makes it easier to preserve instrument transients and overtone details, improving the clarity of the overall audio texture.
• Lyrics-Audio Time Alignment Mechanism: During training and generation, Khala strengthens the temporal alignment constraints between lyrics and acoustic features. By pinning lyrics, beats, and vocal fluctuations onto the same timeline, it reduces lyric misalignment and accent drift, improving the naturalness of the singing.

How to Use Khala

• Visit the Project Repository: Search for the Khala project homepage on GitHub or Hugging Face to obtain the code and model weights.
• Prepare the Hardware Environment: Ensure you have a local GPU with 24GB VRAM level (RTX 4090 recommended).
• Install Dependencies: Follow the deployment instructions in the repository to configure the Python environment and related dependency libraries.
• Load Model Weights: Download and load the pretrained model weights into the local environment.
• Input Condition Generation: Use text prompts or lyric files as input conditions to call the model and generate a complete song.
• Export and Post-Processing: Export the generated audio, which can be further mixed or mastered to improve the finished quality.

Khala's Core Advantages

• Open Source and Free: The code and model weights are fully public, can be deployed locally, and avoid copyright and data security risks.
• Clear Vocals: Compared with traditional open-source models, the vocal articulation is more stable, the AI flavor is lighter, and the listening experience is closer to real human singing.
• High Instrument Separation: The 64-layer acoustic token structure makes each instrument's transients and overtones clearer, so they are less likely to blur together.
• Accurate lyric rhythm alignment: Through a timeline alignment mechanism, it significantly reduces dropped words, reversed words, and beat misalignment issues.
• Academic backing: Jointly developed by the Central Conservatory of Music and Tsinghua University, combining music professionalism with engineering capability.

Khala's project address

• GitHub repository：https://github.com/Khala-Music-AI/Khala
• HuggingFace model library：https://huggingface.co/liujiafeng/Khala-MusicGeneration-v1.0
• arXiv technical paper：https://arxiv.org/pdf/2605.01790

Comparison of Khala with similar competitors

Khala's application scenarios

• Music creation prototype: Musicians can use the model to quickly generate demos and verify the effect of matching melodies with lyrics.
• Academic research: Researchers in music technology and audio AI can use it for algorithm experiments and paper reproduction.
• Independent developer integration: Developers can build on the open-source code for secondary development and embed it into their own music creation tools or platforms.
• Copyright-sensitive scenarios: Enterprises or institutions can deploy it locally and privately to avoid copyright disputes with commercial platforms.
• Music education: Teachers and students at institutions can use it for teaching demonstrations and to study the principles of AI-assisted composition and acoustic modeling.