T3PO - NetEase Youdao's open-source streaming simultaneous translation model

What is T3PO

T3PO is NetEase Youdao's open-source streaming simultaneoustranslationmodel, with the full name "simulTaneous Translation via pareTo Policy Optimization". T3PO focuses on translating while speaking, dynamically judging whether the context is sufficient. If it is enough, it starts translating immediately; if not, it waits a little longer, using a Pareto approach to balance quality and latency. T3PO can be connected after R2T2 to form a speak-listen-translate pipeline, with optimizations for scenarios such as speaker self-corrections, financial report numbers, and filtering English verbal tics.

T3PO's main features

• Streaming simultaneous translation: when the source language keeps coming in, it produces translations while receiving instead of waiting for the whole sentence to finish.
• Read/write decision: if the context is sufficient, it WRITEs and outputs a translation; if not, it READs and keeps waiting, actively balancing "fast" and "accurate".
• Adjustable quality-latency: through a Pareto approach, translation quality and latency are optimized separately, and the same policy can lean toward low latency or high fidelity.
• Closed-loop speech simultaneous translation: after connecting to R2T2, the incremental recognized text that has already been "locked in" is used as the source language, forming "speaking while listening while translating".
• Adaptation to real spoken scenarios: aimed at scenarios such as speaker self-corrections, financial report numbers, and English verbal tics, it adopts processing strategies closer to human simultaneous interpreting.

T3PO's technical principles

• Task modeling: T3PO treats simultaneous translation as an incremental decision process: at each step, the model observes "the source-language prefix heard so far + the target-language prefix generated so far", then decides whether to READ and continue waiting for source language or WRITE the next target word, until the output ends.
• Separating quality and latency: translation quality and response latency are measured separately, and an operating point is chosen on the Pareto frontier; the same model can serve both needs: "better slow but accurate" and "produce a draft first, then revise".
• Pareto DPO optimization: on the training side, preference optimization methods such as Pareto DPO/PDPO are used, making the policy prefer read/write actions with "higher quality at the same latency, lower latency at the same quality".
• Incremental autoregressive decoding: at inference time, WRITE autoregressively produces target tokens, while READ consumes more source-language context; the target prefix simultaneously serves as "already translated content" and "subsequent constraints", and output is triggered only after the source prefix is semantically sufficient, reducing mistranslation of half-sentences.
• Cascading with ASR: T3PO itself is more text-to-text simultaneous translation; after the front end is connected to append-only ASR such as R2T2, each time the recognition end emits a stable segment of text, the translation end immediately READs/WRITEs.

Follow WeChat and reply "open source" to join theAI open-source project communication group

How to use T3PO

• Enter the repository: visit the GitHub repository https://github.com/netease-youdao/Confucius4-T3PO。
• Prepare the environment: follow the README to create an isolated Python/CUDA environment and install dependencies.
• Download weights: pull from Hugging Face or ModelScope Confucius4-T3PO and the corresponding backend model.
• Quick validation: run the official example or Web UI to confirm that it can output READ/WRITE streaming translations.
• Connect speech recognition: start the R2T2 WebSocket and use the stable incremental text as T3PO's source-language input.
• Adjust latency tiers: Adjust read/write strategies and chunk/wait-related parameters based on quality–latency requirements.
• Do robustness testing: Focus on testing self-corrections, numbers, proper nouns, Chinese-English mixing, and long-sentence segmentation.
• Deploy to production: Package into a streaming service, and monitor first-token latency, retractions/stalls, GPU memory, and concurrency.

T3PO's Core Advantages

• Low-latency streaming output: It does not wait for the entire sentence to finish; it produces translations while receiving the source language, approaching the rhythm of human simultaneous interpreting.
• Quality–latency trade-off: Uses a Pareto approach to trade off between "faster" and "more accurate" as needed.
• Adaptive read/write decisions: It waits when context is insufficient, and only starts translating when there is enough, reducing mistranslation of half-sentences and back-and-forth rework.
• Adapted to real spoken language: It is more stable in scenarios such as self-corrections, financial-report numbers, Chinese-English mixing, and verbal tics, approaching a practical simultaneous interpreting experience.
• Easy to connect to speech recognition: When chained with R2T2, it can directly form a speech simultaneous interpreting pipeline that speaks, listens, and translates at the same time.

T3PO's Project Address

• GitHub repository：https://github.com/netease-youdao/Confucius4-T3PO
• HuggingFace model library：https://huggingface.co/netease-youdao/Confucius4-T3PO

Comparison of T3PO with Similar Competing Products

T3PO's Application Scenarios

• Multinational conferences and product launches: Generate Chinese-English subtitles in real time, letting cross-language attendees listen and read the translation at the same time.
• Earnings calls and investment research roadshows: Prioritize ensuring the accuracy of numbers, units, and proper nouns to avoid investment misjudgments caused by streaming mistranslation.
• Cross-border customer service and call centers: Provide low-latency speech translation for agents and customers, and support private deployment to meet compliance requirements.
• Online education and classroom simultaneous interpreting: Translate teachers' lessons into students' languages in real time, and combine glossaries to improve the accuracy of course-specific proper nouns.
• Exhibitions, travel, and offline cross-language communication: Provide portable subtitles or headset simultaneous interpreting in noisy, mobile, and unstable-network environments, facilitating instant communication.