What is Audio Decomposition
Audio Decomposition is an audio processing technology that separates individual notes and instruments in music based on Fourier transform and envelope matching, enabling conversion from music to sheet music. The Audio Decomposition open-source project was launched by Matthew Bird. It does not require external instrument separation libraries and automatically identifies and separates different instrument sounds in music, helping users better understand and reproduce sheet music.
Main features of Audio Decomposition
- Audio source separation: Separate the sounds of different instruments from mixed music.
- Music to sheet music: Convert music files into sheet music, helping users identify pitches and chords.
- Audio analysis: Analyze notes and instruments in music, providing an in-depth understanding of musical structure.
- Open-source tool: As an open-source project, it supports users in freely using and modifying the code to adapt to different needs.
Technical principles of Audio Decomposition
- Fourier transform：The program performs a Fourier transform on the music file every 0.1 seconds to generate a spectrogram.Add the Fourier transforms of each instrument to recreate the spectrum of the music.
- Envelope analysis：Divide the sound wave into several blocks and take the maximum value of each block to obtain the envelope.Further optimize the result by finding points where the envelope is lower than the original sound wave and adding new points to define the envelope.
- Decomposition of the envelope：Decompose the envelope of the wave into attack, sustain, and release.
- Waveform classification：Consider the instrument's static decay and dynamic changes, and whether there is a release.
- Band-pass filtering：Perform band-pass filtering on the signal at each note frequency to separate the sound of a specific instrument.
- Cross-correlation and mean squared error (MSE)：Use the cross-correlation of the instrument's attack and release to find the start and end of each note.Calculate the MSE between the instrument wave and the filtered audio to obtain the cost of the instrument.
Project address of Audio Decomposition
- Project official website：matthew-bird.com/blogs/Audio-Decomposition
- GitHub repository：https://github.com/mbird1258/Audio-Decomposition
Application scenarios of Audio Decomposition
- Music education: Separate the sounds of different instruments to help students more intuitively understand harmony and orchestration in musical works.
- Music production: Support producers in extracting individual tracks from complex music for use in mixing or creating new musical elements.
- Sheet music transcription: Automatically convert audio recordings into sheet music, saving the time and cost of manual transcription, especially suitable for classical music works.
- Music analysis: In-depth analysis of the composition of musical works, providing a research tool for music theorists and composers.
- Audio editing: In film and video production, help editors accurately process and adjust audio elements such as dialogue, music, and sound effects.