What is MMSearch
MMSearch is a benchmark for evaluating the capabilities of Large Multimodal Models (LMMs) as AI search engines. It includes an MMSearch-Engine framework and an MMSearch test set, the latter comprising 300 questions spanning 14 subdomains. The MMSearch-Engine framework enhances the search capabilities of LMMs through three stages: question reformulation, web page ranking, and answer summarization. The test set is divided into two major categories, news and knowledge, ensuring that the test content does not overlap with the training data of LMMs, providing a fair evaluation. Experiments show that the GPT-4o model performs excellently on the MMSearch benchmark, surpassing the commercial product Perplexity Pro. The study finds that increasing test-time computation is more effective than scaling up model size.
Main Features of MMSearch
- Multimodal Search Capability Evaluation: MMSearch aims to evaluate the search capabilities of Large Multimodal Models (LMMs) when handling complex queries containing images and text.
- Question Reformulation: Converts the user's original query into a format suitable for search engine processing.
- Web Page Ranking: Selects the most relevant web pages from the results obtained from the search engine.
- Answer Summarization: Extracts and summarizes answers from the selected web page content.
Technical Principles of MMSearch
- MMSearch-Engine Framework：
- Question Reformulation (Requery): Based on LMMs understanding the intent of the user's query and converting it into a format that search engines can understand. If the query contains images, tools such as Google Lens are also used to identify key information in the images.
- Web Page Ranking (Rerank): Uses LMMs to re-rank the results returned by the search engine, determining which web pages are most likely to contain the correct answer.
- Answer Summarization (Summarization): Extracts relevant information from the selected web pages and summarizes it into an answer.
- Dataset Design: MMSearch contains 300 manually collected query instances, covering two major domains, news and knowledge, ensuring no overlap with the training data of LMMs.
- Evaluation Strategy: Evaluates the performance of LMMs through end-to-end tasks and individual tasks (question reformulation, web page ranking, answer summarization).
Project Addresses of MMSearch
- Project Official Website：mmsearch.github.io
- GitHub Repository：https://github.com/CaraJ7/MMSearch
- HuggingFace Model Library：https://huggingface.co/datasets/CaraJ/MMSearch
- arXiv Technical Paper：https://arxiv.org/pdf/2409.12959
Application Scenarios of MMSearch
- Academic Research: Researchers use MMSearch to evaluate and compare the performance of different Large Multimodal Models when handling complex queries containing images and text.
- Search Engine Optimization: Search engine developers use MMSearch's framework and evaluation methods to improve search engine algorithms, better understanding and responding to users' multimodal queries.
- Intelligent Assistants: When developing intelligent assistants or chatbots, MMSearch enhances the assistant's ability to understand and respond to users' input image and text information.
- Content Recommendation Systems: Content recommendation platforms use MMSearch's technical principles to more accurately analyze users' query intent, thereby providing more relevant recommended content.
- Education and Training: Educational tools integrate MMSearch's evaluation methods to design courses and training materials, helping students better understand the concepts of multimodal information retrieval.