SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

AudioFly - iFlytek's Open-Source Text-to-Sound-Effect Model

Author: AI小集 · Original source: AI工具集

What is AudioFly

AudioFly is an open-source AI model from iFlytek for generating sound effects from text. The model uses a latent diffusion model architecture, has 1 billion parameters, and was trained on a large number of open datasets (such as AudioSet, AudioCaps, TUT) and internal proprietary data. AudioFly can generate high-quality audio based on text descriptions, with a sampling rate of up to 44.1kHz, and the generated sound effects closely match the text descriptions. The model performs excellently in both single-event and multi-event scenarios, and achieves outstanding performance on the AudioCaps dataset, surpassing previous audio generation models. AudioFly is suitable for fields such as short video dubbing and audio story generation, bringing unlimited possibilities to sound creation.

AudioFly

AudioFly's Main Features

  • Text-to-Sound-Effect Generation: Generates corresponding sound effects based on the text descriptions entered by the user. For example, when entering "thunder rumbling in the distance," the model can generate the corresponding thunder sound effect.
  • High-Quality Audio Output: The generated audio has a sampling rate of 44.1kHz, with clear sound quality, suitable for a variety of application scenarios.
  • Multi-Scenario Support: Supports sound effect generation in single-event (such as "dog barking") and multi-event (such as "dog barking and wind sound") scenarios, and can accurately reflect the described content.
  • Efficient Generation: Based on an advanced diffusion model architecture, the generation process is efficient and can quickly respond to user needs.

AudioFly's Technical Principles

  • Latent Diffusion Model (LDM) Architecture: AudioFly uses a latent diffusion model architecture, a generative model based on deep learning. The model generates the target audio by gradually removing noise, similar to the diffusion process in image generation.
  • Large-Scale Data Training: The model is trained on a large number of open datasets (such as AudioSet, AudioCaps, TUT) and internal proprietary data. The data covers various sound effects and scenarios, enabling the model to generate diverse sound effects.
  • Feature Alignment: By optimizing the model's training objective, it ensures that the generated audio is highly consistent with real audio in terms of features, while also closely aligning with the text description in content.

AudioFly's Project Address

  • ModelScope Community：https://modelscope.cn/models/iflytek/AudioFly

AudioFly's Application Scenarios

  • Short Video Dubbing: Quickly generates matching sound effects for short videos, enhancing the appeal and immersion of the videos.
  • Audio Story Creation: Generates sound effects based on the text content, enhancing the atmosphere and emotional expression of the story.
  • Film and Television Sound Effect Production: Assists film and television production teams in quickly generating the required sound effects, improving production efficiency.
  • Game Sound Effect Design: Generates real-time sound effects for game scenes, enhancing players' immersion and experience.
  • Advertising and Marketing: Generates customized sound effects for advertising videos or audio content, enhancing the appeal and memorability of advertisements.