What is TextHarmony
TextHarmony is a multimodal generation model jointly launched by East China Normal University and ByteDance, specializing in understanding and generating visual text. The model is based on Slide-LoRA technology, dynamically aggregating modality-specific and modality-agnostic LoRA experts, partially disentangling the multimodal generation space, and coordinating the generation of vision and language within a single model instance. TextHarmony achieves a more unified generation process between visual and language modalities. The research team has introduced the high-quality image caption dataset DetailedTextCaps-100K, synthesized based on an advanced closed-source MLLM, further enhancing the model's visual text generation capabilities.
Main features of TextHarmony
- Visual text understanding: TextHarmony can understand textual information in images and perform tasks such as scene text detection, recognition, document understanding, visual question answering (VQA), and key information extraction (KIE).
- Visual text generation: The model generates images based on text descriptions and renders accurate and coherent text in the generated images.
- Visual text editing: TextHarmony replaces or renders text at specific positions in an image while maintaining background consistency.
- Visual text perception: The model has basic optical character recognition (OCR) capabilities and can detect and recognize text in images.
Technical principles of TextHarmony
- Slide-LoRA: A method that dynamically aggregates modality-specific and modality-agnostic LoRA (Low-Rank Adaptation) experts, partially disentangling the multimodal generation space, and making the model more coordinated when generating visual and language modalities within a single instance.
- Multimodal pretraining: TextHarmony is pretrained on image-text corpora containing rich text to learn to generate multimodal outputs.
- Comprehensive fine-tuning: Building on pretraining, TextHarmony is fine-tuned on text-centric tasks to improve the model's capabilities in multimodal generation and following human instructions.
- High-quality dataset: The DetailedTextCaps-100K dataset is created, providing detailed image-caption pairs that focus on visual and textual elements in image generation tasks, significantly improving the quality of image generation.
- Model architecture: TextHarmony's architecture includes a vision encoder, an LLM (large language model), and an image decoder. The components work together to generate visual and textual content. It maximizes conditional probability to achieve the generation of interleaved token sequences of text and images.
TextHarmony project address
- GitHub repository：https://github.com/bytedance/TextHarmony
- arXiv technical paper：https://arxiv.org/pdf/2407.16364
Application scenarios of TextHarmony
- Document analysis: TextHarmony is used to automatically process and understand textual content in documents, such as automatically recognizing and extracting key information from documents, improving the efficiency of document management.
- Scene text recognition: In fields such as urban surveillance and autonomous driving, TextHarmony recognizes and understands textual information in scenes such as street views, billboards, and traffic signs.
- Visual question answering (VQA): TextHarmony can answer questions about image content and has broad applications in education, entertainment, and intelligent assistants.
- Image editing and enhancement: In design and media production, TextHarmony adds or modifies text in images as needed, improving the flexibility of creative work.
- Information retrieval: TextHarmony helps develop smarter search engines that understand text content in images to provide more accurate search results.