What is NoteLLM
NoteLLM is a multimodal large language model framework launched by Xiaohongshu for note recommendation. NoteLLM is based on compressed embeddings of generated notes and automatically generated tag categories, using the powerful semantic understanding capabilities of large language models (LLMs), combined with contrastive learning and instruction fine-tuning techniques, to improve the accuracy and relevance of note recommendation. NoteLLM-2 introduces multimodal input on the basis of NoteLLM, based on an end-to-end fine-tuning strategy, combining a visual encoder and an LLM to solve the problem of visual information being overlooked. NoteLLM-2 proposes two mechanisms, multimodal in-context learning (mICL) and late fusion, to further enhance multimodal representation capabilities and significantly improve the performance of multimodal recommendation tasks. The framework demonstrates strong recommendation capabilities on the Xiaohongshu platform and has been applied in the actual recommendation system.
Main features of NoteLLM
- Automatic tag category generation: Generate tags and categories for notes to enhance the quality of note embeddings.
- Improve user experience: Based on more accurate recommendations, increase user engagement and satisfaction on the platform.
- Multimodal note recommendation: Combine text and image information to generate more comprehensive note representations and improve the accuracy and relevance of multimodal recommendation.
- Solve the problem of visual information being overlooked: Based on multimodal in-context learning (mICL) and late fusion mechanisms, enhance the representation capability of visual information.
Technical principles of NoteLLM
- Note Compression Prompt: Design specific prompt templates to compress note content into a special token while generating tags and categories.
- Contrastive Learning: Build related note pairs based on the co-occurrence mechanism in user behavior data, train the model with contrastive learning, and enhance the semantic representation of note embeddings.
- Instruction Tuning: Based on instruction fine-tuning, enable the LLM to better understand task requirements and generate high-quality tags and categories.
- Multimodal in-context learning (mICL): Separate multimodal content into two parts, visual and textual, compress them into two modality compression tokens respectively, and balance attention between modalities based on contrastive learning.
- Late Fusion: Directly fuse visual information at the output stage of the LLM, retaining more original visual information and avoiding the loss of visual information caused by early fusion.
- End-to-end fine-tuning: Combine any existing LLM and visual encoder, based on end-to-end fine-tuning, to customize an efficient multimodal representation model without the need for pretraining alignment.
NoteLLM project address
- GitHub repository：https://github.com/Applied-Machine-Learning-Lab/NoteLLM
- arXiv technical paper：
- NoteLLM：https://arxiv.org/pdf/2403.01744
- NoteLLM2：https://arxiv.org/pdf/2405.16789
Application scenarios of NoteLLM
- Personalized note recommendation: Based on user interests and behavior, accurately recommend related content from massive notes and improve the user discovery experience.
- Cold-start note recommendation: Help newly published notes quickly gain exposure and make recommendations based on content similarity.
- Tag and category generation: Automatically generate tags and categories related to note content, improve content searchability, and help users find content they are interested in faster.
- Multimodal content recommendation: Process text and image information, generate more comprehensive note representations, and improve the accuracy and relevance of multimodal recommendation.
- Content creation assistance: Provides creators with creative inspiration and suggestions, such as keywords, tags, and related note recommendations, to assist content creation.