Seamless is a series of foundational models open-sourced by Meta (facebookresearch), aimed at high-quality speech and text translation, with the goal of making cross-lingual communication more natural and authentic.
I. Model Family and Uses SeamlessM4T is the foundational all-in-one large-scale multilingual multimodal machine translation model, supporting speech and text translation for about 100 languages. The tasks it supports include: speech-to-speech translation (S2ST), speech-to-text translation (S2TT), text-to-speech translation (T2ST), text-to-text translation (T2TT), and automatic speech recognition (ASR). Its v2 version adopts the new UnitY2 architecture, improving quality and reducing inference latency compared with v1 on speech generation tasks. SeamlessExpressive is a speech-to-speech translation model that can capture prosodic features such as speaking rate and pauses while preserving the speaker's voice style and maintaining high content translation quality. SeamlessStreaming is a streaming translation model that supports speech input and can output either speech or text, supporting simultaneous interpretation and streaming ASR. Seamless is a unified model for expressive streaming speech-to-speech translation, essentially replacing SeamlessStreaming's non-expressive vocoder vocoder_v2 with the expressive vocoder vocoder_pretssel.
II. Main Capabilities The repository also open-sources the Conformer-based W2v-BERT 2.0 speech encoder, which is a core component of the Seamless models. It also includes evaluation work for multimodal translation, such as BLASER 2.0 (a model-based evaluation metric supporting speech and text without the need for intermediate ASR), whose underlying embedding space is SONAR. SONAR is a sentence-level multimodal, language-agnostic representation space that provides text and speech encoders for multiple languages. SimulEval is used to evaluate simultaneous interpretation models and provides a streaming inference backend. The project has also released the expressive speech datasets mExpresso and mDRAL (between English and French, German, Italian, Mandarin Chinese, and Spanish), as well as the expressive speech alignment dataset SeamlessAlignExpressive (the first large-scale multilingual expressive translation audio alignment collection of more than 11,000 hours).
III. Usage Prerequisites: fairseq2 is required, and its precompiled packages are only provided for Linux x86-64 and Apple silicon Mac; also depends on libsndfile; Whisper used in evaluation is installed automatically, while Whisper requires ffmpeg installed on the system. Inference can be run through the CLI in the repository root directory; for the specific commands and the list of supported languages, see the inference README. unity.cpp can also be used to run SeamlessM4T's S2TT/ASR natively with GGML (C tensor library), without Python. SeamlessExpressive requires first obtaining the model files through an application form (after approval, the download link is sent by email), and its use is subject to a separate license and acceptable use policy. For the evaluation of SeamlessStreaming and Seamless, see the Streaming Evaluation README; the CLI's --no-scoring option can skip scoring and perform inference only. The SeamlessStreaming demo can be run by duplicating the Hugging Face Space, or cloned locally; the Gradio demos for SeamlessM4T and SeamlessExpressive also support local startup. The models are provided on Hugging Face in the form of Model Cards, and SeamlessM4T can also be used in the Transformers library. In addition, a NeurIPS 2023 Seamless EXPO tutorial notebook is provided.
IV. Applicable Scenarios Suitable for multilingual speech and text translation research, simultaneous interpretation system development, streaming speech recognition, translation experiments that preserve speech style and prosody, and multimodal translation evaluation and dataset construction.
V. Limitations and License Official development stopped around November 2024; the pinned fairseq2 0.2.x dependency cannot be installed on Python 3.12 and above. Non-generative components are under the MIT license (such as the text-only portion of mExpresso, the UnitY2 forced alignment extractor, the etox-based speech toxicity tool, and MuTox). The SeamlessM4T (v1 and v2) and SeamlessStreaming models are under the CC-BY-NC 4.0 license. The Seamless and SeamlessExpressive models are under the Seamless license.
