SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

SAM Audio - Meta's Open-Source Audio Segmentation Model

Author: AI小集 · Original source: AI工具集

What Is SAM Audio

SAM Audio is an open-source audio segmentation model from Meta that can isolate specific sounds from complex audio mixtures using multimodal prompts such as text, vision, and time segments. Its core technology is Perception Encoder Audiovisual (PE-AV), which is based on Meta's open-source Perception Encoder model and can fuse audiovisual information and perform precise temporal annotation to achieve high-precision audio separation. Users can use SAM Audio through simple text descriptions (such as "guitar sound"), by clicking on the sounding object in a video, or by marking the time range in which the sound occurs.

SAM Audio

SAM Audio's Main Features

  • Multimodal-Prompted Audio Separation: Isolate specific sounds from complex audio mixtures through text descriptions, visual selections (objects in videos), or time segment markers.
  • Support for Multiple Audio Tasks: Can handle multiple audio separation tasks such as speech, music, and general sound effects, making it suitable for different scenarios.
  • Reference-Free Audio Evaluation: Provides SAM Audio Judge, an objective evaluation model that requires no reference audio and assesses audio separation quality from a perceptual perspective.
  • Real-World Benchmark Testing: Releases SAM Audio-Bench, the first real-world audio separation benchmark, covering multiple audio domains and prompt types and supporting reference-free evaluation.
  • Efficient Real-Time Processing: Runs faster than real time, with a real-time factor of approximately 0.7, making it suitable for large-scale audio processing.
  • Advancing Accessibility Technology: Collaborates with relevant institutions to explore applications in accessibility technology, such as hearing aid integration, promoting inclusive development of audio AI.

SAM Audio's Technical Principles

  • Perception Encoder Audiovisual Module (PE-AV): Based on Meta's open-source Perception Encoder model, PE-AV can extract frame-by-frame video features and align them with audio representations, fusing audiovisual information to provide precise temporal annotation and semantically rich feature representations for audio separation.
  • Flow-Matching Diffusion Transformer Architecture: Adopts a generative modeling framework that encodes mixed audio and prompt information into a shared representation, generates the target track and the remaining track, and supports prompt inputs from multiple modalities.
  • Large-Scale Multimodal Data Training: Trained by fusing real and synthetic mixed audio data, covering multiple sound events such as speech, music, and general sound effects, and combining advanced audio synthesis strategies to improve the model's robustness and generalization ability.
  • Reference-Free Audio Evaluation Model: SAM Audio Judge evaluates audio quality based on perceptual dimensions, requires no reference track, and provides objective evaluation standards that are closer to the human listening experience.
  • Real-World Benchmark Testing: SAM Audio-Bench, as the first real-world audio separation benchmark, covers multiple audio tasks and modality prompts, supports reference-free evaluation, and ensures the model's effectiveness in practical applications.

SAM Audio's Project Address

  • Project Website：https://ai.meta.com/samaudio/
  • Github Repository：https://github.com/facebookresearch/sam-audio

SAM Audio's Application Scenarios

  • Audio Cleanup and Background Noise Removal: Remove unwanted background noise from audio files, such as dog barking or traffic noise in podcast recordings, improving audio quality.
  • Creative Media Production: Help creators extract specific instrument sounds from songs, separate vocals, or remix audio, enhancing creative expression.
  • Accessibility Technology: Collaborate with hearing aid manufacturers to explore applications in hearing assistance devices, helping people with hearing loss better understand audio content.
  • Video Editing: In video production, separate the sound of specific objects through visual cues, for example, clicking on a guitarist in the video to extract the guitar sound, improving the flexibility of video editing.
  • Audio Analysis and Research: Provide tools for audio analysis, helping researchers separate and study specific sound events, suitable for fields such as music analysis and sound ecology.