What is Poetry2Image
Poetry2Image is an iterative correction framework designed for Chinese classical poetry image generation, proposed by Harbin Institute of Technology. The framework enhances the consistency between poetry and images through an automated feedback and correction loop, effectively capturing the semantics and artistic essence of poetry. It addresses the common issues of missing key elements or semantic confusion in text-to-image generation models when processing Chinese classical poetry. Poetry2Image can improve the element completeness and semantic consistency of image generation, achieving an average element completeness of 70.63% and semantic consistency of 80.09% when combined with five popular image generation models.
Main Features of Poetry2Image
- Automated feedback and correction: Poetry2Image uses an external poetry dataset to establish an automated feedback and correction loop, which can enhance the alignment between poetry and images and improve the quality and accuracy of generated images.
- Reduced manual intervention: The framework reduces the extensive manual intervention and professional knowledge required in traditional methods, making the image generation process more efficient.
- Improved efficiency and quality: Compared with traditional fine-tuning methods, Poetry2Image significantly reduces training cost and time while maintaining the quality of generated images.
- Search and translation: The system searches the poetry database for the ancient poem provided by the user and finds its modern Chinese translation and appreciation.
- Generate initial image: Based on the modern Chinese translation of the poem, the system generates a preliminary image.
- Extract key elements: The system uses a large language model to identify and extract key elements from the poem.
- Image correction: The system checks whether the generated image contains all these key elements and, when necessary, proposes modification suggestions, such as adding missing elements or adjusting the positions of elements.
- Iterative optimization: If the elements in the image are incomplete or incorrect, the system generates the image again based on the suggestions, and this process repeats until the image can accurately reflect the artistic conception of the poem.
Technical Principles of Poetry2Image
- Recognition and correction of image elements: Based on an open-vocabulary detector (OVD), it identifies element information in the image, and then provides modification suggestions through an LLM. These suggestions are presented in the form of bounding boxes in the image, guiding the image editing model to edit the initial image.
- Compatibility and training cost: Poetry2Image imposes no restrictions on the text-to-image generation model used for initial image generation, and the iterative correction operation eliminates the need for additional training cost, while the automated image generation and feedback process significantly reduces manual annotation.
Project Address of Poetry2Image
- Github Repository：https://github.com/prajwalppv/Poetry2Image
- arXiv Technical Paper：https://arxiv.org/pdf/2407.06196v1
Application Scenarios of Poetry2Image
- Dissemination of ancient poetry culture: Poetry2Image promotes the dissemination of ancient poetry culture by generating images that match the artistic conception of ancient poetry, allowing traditional culture to be understood and appreciated by modern people in a more vivid form.
- Assistance in artistic creation: It can help artists and designers quickly generate visual elements that match ancient poetry during the creative process, providing inspiration and materials.
- Education and learning: In the field of education, Poetry2Image can serve as a teaching tool to help students better understand and remember ancient poetry, enhancing the learning experience through the combination of images and poetry.
- Research on Image Generation Technology: Poetry2Image provides a research platform for exploring and improving text-to-image generation technology, especially when dealing with Chinese ancient poetry that has rich cultural connotations and complex semantics.