MMAE - An Audio Editing Evaluation Benchmark Jointly Launched by Tencent Hunyuan and Universities

What is MMAE

MMAE (Massive Multitask Audio Editing Benchmark) is the first large-scale multitask evaluation benchmark for general instruction-based audio editing, jointly launched by Shanghai Jiao Tong University, Shanghai Innovation Institute, Nanyang Technological University, the Tencent Hunyuan team, and other institutions. The benchmark contains 2,000 real-world audio editing tasks and 17,741 fine-grained rubric evaluation items, systematically covering 7 audio modalities, 6 task difficulty levels, and 8 categories of editing operations, establishing a standardized and interpretable evaluation paradigm for next-generation intelligent audio editing systems.

Main Features of MMAE

• Real-world task library: Provides 2,000 high-fidelity audio editing samples from real-world scenarios rather than synthetic data, ensuring that evaluation stays close to practical applications.
• Three-dimensional systematic classification: Establishes an orthogonal classification system of Modality (7 modalities: sound / music / speech and pairwise/triple mixtures), Complexity (6 difficulty levels: single → multi-part → multi-instruction → multi-audio → multi-round → multi-hop), and Operation (8 categories of operations: local add/delete/modify/query attribute editing + global background/foreground/attribute editing).
• Fine-grained Rubric evaluation: Designs on average about 9 atomic, verifiable multiple-choice-style rubrics for each sample, evaluating from two independent dimensions: instruction following and "context consistency".
• Multidimensional quantitative metrics: Outputs IFR, CR, and EMR, providing a comprehensive characterization from average capability to perfect execution rate.
• Automated evaluation pipeline: Uses the Qwen3-Omni multimodal large model as the judge, supporting automatic and reproducible scoring of model outputs.

Technical Principles of MMAE

• Human-Agent collaborative annotation: Adopts a five-stage data construction process—expert brainstorming to collect scenarios → build taxonomy and evaluation paradigm → instruction-centric data collection and dynamic balancing → human-machine collaborative rubric annotation → cross-review quality check.
• Rubric-Based evaluation paradigm: Decomposes open-ended free-form editing tasks into 17,741 atomic, orthogonal, objective multiple-choice questions, with each rubric assessing only one indivisible attribute, ensuring interpretable evaluation and avoiding the ambiguity of signal-level metrics.
• Two-dimensional orthogonal evaluation: The IFR dimension assesses whether the model precisely performs the modifications required by the instruction, while the CR dimension assesses whether the parts not involved in the instruction are strictly preserved. The combination of the two effectively blocks speculative strategies of "only modifying without preserving" or "only preserving without modifying."
• Stable judging mechanism: Uses Qwen3-Omni as the external judging model. Each rubric is independently queried 3 times with majority voting, and the option order is randomly shuffled each time to eliminate position bias.

Follow WeChat and reply "Open Source" to join theAI Open Source Project Exchange Group

How to Use MMAE

• Deploy the judging model: Clone the official Qwen3-Omni repository and configure the environment, start the vLLM service, and refer to the script to launch two tensor-parallel=4 instances on 8 GPUs, listening on ports 8001 and 8002 respectively.
• Prepare prediction results: Run the audio editing model to be tested on the MMAE benchmark metadata, and append the output audio paths in chatml format as assistant Reply, and save as a JSON file.
• Run automatic evaluation: Execute the python -m eval.score command, pass in the prediction file path, the judge model API address, the audio root directory, and the concurrency count to automatically generate metrics such as IFR, CR, and EMR.

MMAE's Core Advantages

• Real-Data Driven: All samples are sourced from real-world audio, undergoing multiple rounds of detailed revision and independent review, avoiding the distribution shift caused by synthetic data.
• Errors Can Be Located: Compared with coarse-grained metrics such as FAD and CLAP, which can only give an overall score, MMAE's rubric system can precisely diagnose where a model goes wrong and provide a clear roadmap for diagnosing model capabilities.
• Anti-Gaming Design: By evaluating both IFR and CR and introducing the EMR metric, models are forced to strike a balance between accurate modification and preserving the original content, and cannot game the score through a single strategy.
• Cross-Modal Unification: For the first time, sound, music, speech, and their mixed scenarios are incorporated into the same evaluation framework, solving the problems of highly fragmented and strictly domain-limited prior benchmarks.
• Full Difficulty Coverage: From the simplest single-step single-element editing to complex tasks requiring multi-hop reasoning and multi-round context dependence, it fully covers the three layers of model capability in the cognitive pipeline: perception, reasoning, and generation.

MMAE's Project Address

• GitHub Repository：https://github.com/ddlBoJack/MMAE
• arXiv Technical Paper：https://arxiv.org/pdf/2606.07229

Comparison of MMAE with Similar Competitors

MMAE's Application Scenarios

• Audio Editing Model R&D: Provide standardized capability evaluation and iterative diagnostic evidence for models such as Step-Audio-EditX, Ming-UniAudio, and Audio-Omni.
• Multimodal Large Model Audio Module Evaluation: Evaluate the cross-modal editing robustness of a unified audio understanding/generation/editing system.
• Academic Research Benchmark: Serve as a public testing platform for fairly comparing different audio editing architectures, promoting standardization in the field.
• Industrial Quality Inspection: Audio content production platforms can use the MMAE framework to build internal evaluation systems and select the optimal editing model for deployment.