Qwen-Image-Bench - A Text-to-Image Model Evaluation Benchmark Introduced by Tongyi Qianwen

What is Qwen-Image-Bench

Qwen-Image-Bench is introduced by the Tongyi Qianwen teamText-to-imagemodel evaluation dataset, with a scale of 1k test samples, covering Chinese and English bilingual prompts, supporting multi-dimensional evaluation of the generation effects of various text-to-image models, and is open-sourced under the Apache-2.0 license. It provides a standardized and reproducible evaluation framework for image generation models, covering core capability dimensions such as text rendering, image editing, and general generation, helping developers and researchers objectively compare the actual performance of different models.

Main Features of Qwen-Image-Bench

• Chinese-English bilingual evaluation: Test samples cover Chinese and English prompts, enabling comprehensive assessment of the model's multilingual image generation capabilities.
• Multi-dimensional capability evaluation: Supports comprehensive evaluation of text-to-image models across dimensions such as text rendering, image editing, general generation, and semantic consistency.
• Standardized testing process: Provides unified evaluation scripts and data formats to ensure fair comparison between different models.
• Open-source dataset: 1k carefully designed test cases, covering a variety of complex scenarios and fine-grained tasks.
• Automated scoring support: Supports automated evaluation combined with multiple benchmark metrics (such as GenEval, DPG, GEdit, etc.).

How to Use Qwen-Image-Bench

• Clone the repository locally: Visit the GitHub repository QwenLM/Qwen-Image-Bench, use git clone to download the project code to the local environment..
• Install the dependency environment: According to the requirements.txt or documentation in the repository, install Python dependency libraries (such as PyTorch, Diffusers, Transformers and other tools required for image generation and evaluation).
• Prepare the model to be evaluated: Configure the text-to-image model to be tested, supporting local loading of model weights (such as Qwen-Image, FLUX, Stable Diffusion, etc.), or access to remote model services via API.
• Load the evaluation dataset: Load the 1k Chinese-English bilingual test samples provided by Qwen-Image-Bench into the evaluation process. The dataset contains multi-dimensional prompts such as general generation, text rendering, and image editing.
• Execute batch image generation: Run the inference script. The model generates corresponding images one by one according to the text prompts in the dataset. It is recommended to unify the output resolution (such as 1024×1024) and inference parameters to ensure evaluation consistency.
• Run the automated evaluation script: Call the evaluation tools built into the repository to automatically score the generated results across dimensions such as text rendering accuracy, semantic consistency, image quality, and editing fidelity.
• Output and compare evaluation results: Generate a structured evaluation report, view the model's scores across various dimensions, and support horizontal comparative analysis with other models.
• Custom extended evaluation (optional): Custom test cases can be added according to actual needs, or the weights of evaluation metrics can be adjusted to adapt to the evaluation needs of specific business scenarios.

Core Advantages of Qwen-Image-Bench

• Strong specificity for Chinese scenarios: Particularly strengthens the evaluation of Chinese text rendering and understanding of cultural elements, filling the gap of insufficient Chinese coverage in existing benchmarks
• Comprehensive evaluation dimensions: Covers multiple types of tasks such as general image generation, precise image editing, and complex text rendering, and is not limited to a single capability
• Moderate scale and easy to reproduce: 1k samples ensure the representativeness of the evaluation while lowering the reproduction threshold and computational cost
• Good ecosystem compatibility: Natively adapted to models such as Qwen-Image and Qwen-Image-Edit, and can also be used to evaluate third-party text-to-image models
• Friendly open-source license: The Apache-2.0 license allows commercial use and free secondary development

Project address of Qwen-Image-Bench

• Github repository：https://github.com/QwenLM/Qwen-Image-Bench
• HuggingFace model library：https://huggingface.co/datasets/Qwen/Qwen-Image-Bench
• arXiv technical paper：https://arxiv.org/pdf/2605.28091

Comparison of Qwen-Image-Bench with similar competitors

Application scenarios of Qwen-Image-Bench

• Standardized evaluation before model release: Before a text-to-image model is officially launched or open-sourced, use the 1k Chinese-English bilingual test cases of Qwen-Image-Bench to systematically verify the model's generation quality and stability in dimensions such as general generation, text rendering, and image editing, ensuring that the model meets release standards.
• Horizontal capability comparison across multiple models: Horizontally compare the overall performance of different text-to-image models such as Qwen-Image, FLUX, GPT Image 1, SeedDream, and Stable Diffusion, intuitively presenting each model's strengths and weaknesses from scores across dimensions to assist technology selection.
• Specialized testing of Chinese generation capability: Focus on verifying the model's text rendering performance in applications such as Chinese posters, PPTs, e-commerce images, and cultural scenes, and evaluate the model's understanding and visual expression of Chinese semantics, typographic layout, and cultural elements.
• Image editing capability evaluation: Testing model performance in image editing tasks such as style transfer, local modification, text replacement, and object addition/removal, measuring semantic consistency and visual fidelity before and after editing.
• Academic research benchmark citation: Serving as an authoritative benchmark dataset citation for model evaluation in papers, enhancing the credibility and reproducibility of research results and supporting publication of basic research in the field of image generation.