What is TokenFD
TokenFD is a fine-grained image-text alignment foundation model jointly launched by Shanghai Jiao Tong University and Meituan, primarily used for document understanding tasks. Through token-level alignment, it achieves sharing of image tokens and language tokens in the same feature space, supports token-level image-text interaction, and performs outstandingly when processing images containing dense text. To train this model, the team built TokenIT, the industry's first token-level image-text dataset, containing 20 million images and 1.8 billion high-quality Token-Mask pairs, with a data volume far exceeding existing models.
TokenFD's Main Features
- Fine-Grained Image-Text Alignment: TokenFD achieves sharing of image tokens and language tokens in the same feature space, supporting token-level image-text interaction.
- Improving Multimodal Task Performance: TokenFD can significantly improve the performance of multimodal large models in document understanding tasks. For example, in text segmentation tasks, Zero-Shot performance improved by 18.78%; in text understanding tasks, Zero-Shot performance improved by 1.48%; in text retrieval tasks, Zero-Shot performance improved by 50.33%.
- Backbone Adaptation and Extension: TokenFD can directly replace the backbones of other multimodal large models, improving various evaluation benchmarks without additional training. Based on TokenFD as a visual foundation model, TokenVL was further launched, a new multimodal large model image-text alignment pretraining paradigm.
TokenFD's Technical Principles
- BPE Tokenization and Pixel-Level Mask Annotation: TokenFD pioneered BPE tokenization + pixel-level mask annotation technology. Specifically, it splits text into BPE subwords (such as "un-", "-able"), and precisely maps each subword (token) to a local region in the image. This enables the model to better understand textual information in images and supports semantic mapping of "image as text".
- Dataset and Model Training: TokenFD is trained on the independently developed TokenIT dataset. The dataset contains 20 million images and 1.8 billion high-quality Token-Mask pairs, covering multiple text-image types such as natural scenes, documents, and charts. The large-scale fine-grained annotated data provides rich semantic information for the model, enabling outstanding performance in tasks such as document understanding and text segmentation.
TokenFD's Project Address
- Project Website：https://token-family.github.io/project_page/
- Github Repository：https://github.com/Token-family/TokenFD
- HuggingFace Model Library：https://huggingface.co/TongkunGuan/TokenFD
- arXiv Technical Paper：https://arxiv.org/pdf/2503.02304
TokenFD's Application Scenarios
- Document Understanding and Processing: TokenFD can better process document images containing dense text, such as complex structures including tables, formulas, and characters.
- Image Safety Review: TokenFD can be used for image safety review. Through fine-grained image-text alignment, it can more accurately detect violating content in images, such as images containing inappropriate text or symbols.
- Text-Based Image Retrieval: TokenFD breaks through traditional text recognition-based image-text retrieval methods, directly performing similarity matching through the feature space, and supports arbitrary text input for image content search.
- Knowledge Retrieval-Augmented Large Models: TokenFD can be used to enhance knowledge retrieval capabilities. Through fine-grained image-text alignment, it helps large models better understand and generate text content related to images.