What Is HistAgent
HistAgent is an artificial intelligence assistant system designed specifically for historical research, jointly launched by the AI Lab at Princeton University and the Department of History at Fudan University. It addresses challenges in historical research such as multimodal information processing, cross-linguistic analysis, and complex reasoning. HistAgent can process a variety of historical materials, including manuscripts, images, audio, video, inscriptions, and texts, supports 29 ancient and modern languages, and covers multiple historical periods from antiquity to the modern era and content from different regions around the world. On HistBench, a benchmark specifically designed for historical reasoning evaluation, HistAgent significantly outperforms general-purpose large language models and other AI Agents.
HistBench is the world's first AI evaluation benchmark focused on historical research capabilities, jointly developed by the AI Lab at Princeton University and the Department of History at Fudan University. It fills the gap in AI testing in the humanities and promotes systematic testing and capability breakthroughs for AI in the field of history.
HistAgent's Main Features
-
Multimodal Material Processing: HistAgent can process a variety of historical materials such as manuscripts, images, maps, audio, and video. Through its OCR module, it recognizes documents such as manuscripts and inscriptions, supports reverse image search and artifact recognition, and can process audio materials such as historical speeches and interview records.
-
Multilingual Support: HistAgent supports the translation and processing of 29 ancient and modern languages, including classical languages and lesser-known languages. It can translate the surface meaning of texts and can optimize translations based on context.
-
Literature Retrieval and Document Parsing: It supports multi-step web search and page parsing and can retrieve academic websites and historical materials. HistAgent can parse files in multiple formats such as PDF, DOCX, XLSX, and PPTX.
-
Historical Reasoning and Information Integration: HistAgent can assist reasoning by combining historical knowledge, helping researchers sort out clues, integrate information, and form academic judgments. Through the central scheduling module (Manager Agent), it intelligently coordinates various submodules, invokes the corresponding tools according to task requirements, integrates multimodal results, and ultimately outputs complete answers that conform to the standards of the historical discipline.
-
Multi-Agent Collaboration: HistAgent is a multi-agent collaborative system containing multiple submodules. It can simulate the process of historical research, break down complex tasks into different subtasks, and invoke the most appropriate tools according to the requirements of each subtask.
HistAgent's Technical Principles
- Multi-Agent Architecture: HistAgent adopts the design pattern of a Multi-Agent System. It breaks down complex tasks into multiple subtasks and assigns them to different agents for processing. Each agent focuses on a specific task, such as image recognition, language translation, literature retrieval, and so on. In this way, HistAgent can efficiently process multiple types of historical materials and integrate results from different modalities.
-
Task Planning and Execution: The query entered by the user is first decomposed into multiple subtasks, and each subtask is executed by the corresponding agent. The execution results are observed and verified. If the results are unsatisfactory or errors occur, the system will replan and adjust the tasks.
-
Multi-Perspective Analysis and Collaboration: The multi-agent architecture supports analyzing problems from different perspectives. Each agent can independently handle problems in a specific domain, reducing the requirements for memory and prompt length.
-
- Multimodal Processing Technology: HistAgent can handle historical materials in multiple modalities, including text, images, audio, and video. The core of multimodal processing technology lies in converting information from different modalities into a unified semantic representation, facilitating further analysis and reasoning.
-
Visual processing: Images and videos are processed through computer vision (CV) models (such as YOLOv8) to extract key information and convert it into structured descriptions, which are then injected into the context of the large language model.
-
Speech processing: Based on automatic speech recognition (ASR) technology (such as Whisper), audio is converted into text, then processed through the large language model, and finally the results are output through text-to-speech (TTS) technology.
-
- Knowledge enhancement and reasoning: To improve the accuracy and reliability of reasoning, HistAgent adopts knowledge enhancement technology. By storing documents in the knowledge base as vectors (such as ChromaDB), relevant knowledge is dynamically retrieved and injected when processing user queries. This can effectively suppress the hallucination problem of large language models and improve the credibility of output results.
- Tool calling and extension: HistAgent supports dynamic calling of external tools and plugins. Through the tool calling module, HistAgent can call specific APIs or tools according to task requirements, such as literature retrieval and file parsing. This improves the flexibility of the system and supports developers in extending HistAgent's functionality by adding new plugins.
- Memory system: HistAgent's memory system adopts a hybrid memory architecture, including short-term memory and long-term memory. Short-term memory is used to store contextual information of the current task, while long-term memory stores important historical information through a vector database (such as ChromaDB).
HistAgent's project address
- Github repository：https://github.com/CharlesQ9/HistAgent
- arXiv technical paper：https://arxiv.org/pdf/2505.20246
HistAgent's application scenarios
-
Literature retrieval and analysis: Through multi-step web search and page parsing, it retrieves academic websites and historical materials, providing authoritative background information and evidence support.
-
Image and artifact recognition: It can perform reverse image search and artifact recognition, finding sources and supplementing background for historical image materials.
-
Historical reasoning and clue integration: It combines historical knowledge to assist reasoning, helping researchers sort out clues, integrate information, and form academic judgments.
-
History teaching assistance: It provides teachers with rich historical materials and cases, assists teaching design, and improves teaching effectiveness.
-
Cultural heritage protection: Through image recognition and OCR technology, it helps protect and study cultural heritage such as ancient books and inscriptions.
Features of HistBench
- High-quality question bank: The HistBench dataset contains 414 high-quality history questions, which were written by historians and cover multiple levels from basic historical source reading to interdisciplinary in-depth analysis.
- Multilingual and multimodal coverage: The benchmark covers 29 ancient and modern languages, supports multiple types of historical sources such as manuscripts, images, audio and video, and historical artifacts, and realistically simulates historical research scenarios.
- Difficulty grading: Questions are divided into three difficulty levels, from basic information retrieval to complex multimodal historical source processing and interdisciplinary analysis.
- Level 1 (Basic):166 questions, designed by history background assistants, focusing on basic information retrieval and extraction.
- Level 2 (Advanced):172 questions, written by graduate students, requiring a certain degree of difficulty in source material processing or logical reasoning.
- Level 3 (Challenge):76 questions, designed by senior scholars, involving the reading of minor/extinct languages, multimodal historical source processing, and interdisciplinary analysis.
- Broad historical domain coverage: Covering more than 20 historical regions and 36 subfields, including Classical studies, global history, new cultural history, art history, environmental history, history of science, technology, and medicine, etc.